[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480412148 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003156] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008504] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012115] LSM: Security Framework initializing [ 0.013064] Yama: becoming mindful. [ 0.014050] SELinux: Initializing. [ 0.016062] *** VALIDATE selinux *** [ 0.024656] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029542] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032134] *** VALIDATE tmpfs *** [ 0.034073] *** VALIDATE proc *** [ 0.035249] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040042] Spectre V2 : User space: Vulnerable [ 0.041038] Speculative Store Bypass: Vulnerable [ 0.043772] debug: unmapping init [mem 0xffffffff89459000-0xffffffff89460fff] [ 0.045226] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046766] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054294] rcu: Hierarchical SRCU implementation. [ 0.056350] smp: Bringing up secondary CPUs ... [ 0.057605] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.061225] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.245404] node 0 deferred pages initialised in 180ms [ 0.249120] devtmpfs: initialized [ 0.250334] x86/mm: Memory block size: 128MB [ 0.254337] gcov: version magic: 0x41383552 [ 0.257361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.258094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260438] pinctrl core: initialized pinctrl subsystem [ 0.262276] [ 0.262728] ************************************************************* [ 0.265020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267019] ** ** [ 0.269023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.272026] ** ** [ 0.274018] ** This means that this kernel is built to expose internal ** [ 0.277024] ** IOMMU data structures, which may compromise security on ** [ 0.279017] ** your system. ** [ 0.282023] ** ** [ 0.284022] ** If you see this message and you are not debugging the ** [ 0.287022] ** kernel, report this immediately to your vendor! ** [ 0.289031] ** ** [ 0.292026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294021] ************************************************************* [ 0.297782] NET: Registered protocol family 16 [ 0.299463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.303075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.310035] cpuidle: using governor menu [ 0.312197] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.315095] PCI: Using configuration type 1 for base access [ 0.317140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.326099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.328030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.332129] cryptd: max_cpu_qlen set to 1000 [ 0.335332] ACPI: Added _OSI(Module Device) [ 0.337034] ACPI: Added _OSI(Processor Device) [ 0.339026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.341025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.346197] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355042] ACPI: Interpreter enabled [ 0.356095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.358024] ACPI: Using IOAPIC for interrupt routing [ 0.359166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.363506] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.373350] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.375058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.378027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.381095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.386399] acpiphp: Slot [2] registered [ 0.388147] acpiphp: Slot [5] registered [ 0.389115] acpiphp: Slot [6] registered [ 0.390127] acpiphp: Slot [7] registered [ 0.392162] acpiphp: Slot [8] registered [ 0.393130] acpiphp: Slot [9] registered [ 0.394133] acpiphp: Slot [10] registered [ 0.396115] acpiphp: Slot [3] registered [ 0.397093] acpiphp: Slot [4] registered [ 0.399096] acpiphp: Slot [11] registered [ 0.400105] acpiphp: Slot [12] registered [ 0.402075] acpiphp: Slot [13] registered [ 0.403112] acpiphp: Slot [14] registered [ 0.404112] acpiphp: Slot [15] registered [ 0.406104] acpiphp: Slot [16] registered [ 0.408147] acpiphp: Slot [17] registered [ 0.409123] acpiphp: Slot [18] registered [ 0.411129] acpiphp: Slot [19] registered [ 0.412114] acpiphp: Slot [20] registered [ 0.414119] acpiphp: Slot [21] registered [ 0.415106] acpiphp: Slot [22] registered [ 0.417109] acpiphp: Slot [23] registered [ 0.418135] acpiphp: Slot [24] registered [ 0.419128] acpiphp: Slot [25] registered [ 0.421117] acpiphp: Slot [26] registered [ 0.422116] acpiphp: Slot [27] registered [ 0.424102] acpiphp: Slot [28] registered [ 0.425103] acpiphp: Slot [29] registered [ 0.426067] acpiphp: Slot [30] registered [ 0.427130] acpiphp: Slot [31] registered [ 0.429073] PCI host bridge to bus 0000:00 [ 0.430019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.432026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.437028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.440027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.442031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.444204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.447062] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.450640] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.458733] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.465032] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.468020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.470014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473634] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.476905] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.479049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.482541] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.505017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.510477] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.522016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.537025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.545832] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.555035] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.567021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.589021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.598505] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.604020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.609018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.621017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.632291] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.639019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.645017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.657018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.666305] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.672018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.677017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.693031] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.703643] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.708017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.720016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.743021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.756194] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.758330] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.760845] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.763438] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.765184] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.771043] iommu: Default domain type: Passthrough [ 0.773434] SCSI subsystem initialized [ 0.774160] ACPI: bus type USB registered [ 0.776093] usbcore: registered new interface driver usbfs [ 0.778101] usbcore: registered new interface driver hub [ 0.780082] usbcore: registered new device driver usb [ 0.781179] pps_core: LinuxPPS API ver. 1 registered [ 0.783012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.785054] PTP clock support registered [ 0.787033] EDAC MC: Ver: 3.0.0 [ 0.788181] PCI: Using ACPI for IRQ routing [ 0.791126] NetLabel: Initializing [ 0.792018] NetLabel: domain hash size = 128 [ 0.794020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.796093] NetLabel: unlabeled traffic allowed by default [ 0.798168] vgaarb: loaded [ 0.800217] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.801007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.805004] clocksource: Switched to clocksource kvm-clock [ 0.904625] VFS: Disk quotas dquot_6.6.0 [ 0.905737] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.908446] *** VALIDATE ramfs *** [ 0.909865] *** VALIDATE hugetlbfs *** [ 0.911664] pnp: PnP ACPI init [ 0.914531] pnp: PnP ACPI: found 6 devices [ 0.933097] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.936343] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.938373] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.940299] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.942308] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.944447] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.946979] NET: Registered protocol family 2 [ 0.949106] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.953356] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.956653] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.961820] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.965202] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968519] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971809] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.974690] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977875] NET: Registered protocol family 1 [ 0.980851] RPC: Registered named UNIX socket transport module. [ 0.983183] RPC: Registered udp transport module. [ 0.984753] RPC: Registered tcp transport module. [ 0.986654] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.988983] NET: Registered protocol family 44 [ 0.990063] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.991304] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.993205] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.994896] PCI: CLS 0 bytes, default 64 [ 0.996576] Unpacking initramfs... [ 2.452734] debug: unmapping init [mem 0xffff97183cc54000-0xffff97183ffbffff] [ 2.458047] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.460418] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.463764] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.955720] Initialise system trusted keyrings [ 2.957383] Key type blacklist registered [ 2.958875] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.971190] zbud: loaded [ 2.974927] *** VALIDATE nfs *** [ 2.976139] *** VALIDATE nfs4 *** [ 2.977419] pstore: using deflate compression [ 2.980154] Platform Keyring initialized [ 3.103061] NET: Registered protocol family 38 [ 3.105041] Key type asymmetric registered [ 3.106892] Asymmetric key parser 'x509' registered [ 3.108905] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.112455] io scheduler mq-deadline registered [ 3.114342] io scheduler kyber registered [ 3.116029] io scheduler bfq registered [ 3.118508] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.121781] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.124863] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.127935] ACPI: Power Button [PWRF] [ 3.135127] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.142543] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.159992] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.166408] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.179069] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.206791] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.232985] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.238642] Non-volatile memory driver v1.3 [ 3.240397] Linux agpgart interface v0.103 [ 3.273877] virtio_blk virtio1: [vda] 146976 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.277138] vda: detected capacity change from 0 to 75251712 [ 3.291193] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.294709] vdb: detected capacity change from 0 to 1073741824 [ 3.310230] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.314108] vdc: detected capacity change from 0 to 2621440000 [ 3.328099] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.330812] vdd: detected capacity change from 0 to 2621440000 [ 3.344159] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.346930] vde: detected capacity change from 0 to 4294967296 [ 3.360755] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.363487] vdf: detected capacity change from 0 to 4294967296 [ 3.369436] libphy: Fixed MDIO Bus: probed [ 3.382827] usbcore: registered new interface driver usbserial_generic [ 3.385941] usbserial: USB Serial support registered for generic [ 3.388860] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.393912] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.395712] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.398221] mousedev: PS/2 mouse device common for all mice [ 3.401453] rtc_cmos 00:05: RTC can wake from S4 [ 3.404740] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.406115] rtc_cmos 00:05: registered as rtc0 [ 3.411222] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.412524] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.414059] intel_pstate: CPU model not supported [ 3.419925] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.423891] hid: raw HID events driver (C) Jiri Kosina [ 3.425658] usbcore: registered new interface driver usbhid [ 3.427512] usbhid: USB HID core driver [ 3.428899] drop_monitor: Initializing network drop monitor service [ 3.431291] Initializing XFRM netlink socket [ 3.433537] NET: Registered protocol family 10 [ 3.436276] Segment Routing with IPv6 [ 3.437634] NET: Registered protocol family 17 [ 3.439606] mpls_gso: MPLS GSO support [ 3.445129] RAS: Correctable Errors collector initialized. [ 3.446875] AVX version of gcm_enc/dec engaged. [ 3.448161] AES CTR mode by8 optimization enabled [ 3.535923] sched_clock: Marking stable (3535901410, 0)->(4438554428, -902653018) [ 3.538495] registered taskstats version 1 [ 3.539793] Loading compiled-in X.509 certificates [ 3.541450] zswap: loaded using pool lzo/zbud [ 3.565993] Key type big_key registered [ 3.578691] Key type encrypted registered [ 3.581085] ima: No TPM chip found, activating TPM-bypass! [ 3.584118] ima: Allocated hash algorithm: sha1 [ 3.586286] ima: No architecture policies found [ 3.588465] evm: Initialising EVM extended attributes: [ 3.590714] evm: security.selinux [ 3.592111] evm: security.ima [ 3.593564] evm: security.capability [ 3.595143] evm: HMAC attrs: 0x1 [ 3.598212] rtc_cmos 00:05: setting system clock to 2026-08-27 21:20:06 UTC (1787865606) [ 3.605197] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 3.608879] debug: unmapping init [mem 0xffffffff89182000-0xffffffff89458fff] [ 3.623287] Write protecting the kernel read-only data: 28672k [ 3.627041] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 3.630603] debug: unmapping init [mem 0xffffffff88114000-0xffffffff881fffff] [ 3.673254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.683517] systemd[1]: Detected virtualization kvm. [ 3.686120] systemd[1]: Detected architecture x86-64. [ 3.688467] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.718976] systemd[1]: No hostname configured. [ 3.720612] systemd[1]: Set hostname to . [ 3.722335] random: systemd: uninitialized urandom read (16 bytes read) [ 3.724215] systemd[1]: Initializing machine ID from random generator. [ 3.773287] random: ln: uninitialized urandom read (6 bytes read) [ 3.846828] random: systemd: uninitialized urandom read (16 bytes read) [ 3.850163] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.854503] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.858574] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.518236] device-mapper: uevent: version 1.0.3 [ 4.521674] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.325073] virtio_net virtio0 ens2: renamed from eth0 [ 5.355295] random: fast init done [ 5.391036] scsi host0: ata_piix [ 5.412913] scsi host1: ata_piix [ 5.414567] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.417156] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.904307] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.101639] random: crng init done [ 10.102928] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.608437] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.879634] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.095918] SELinux: Disabled at runtime. [ 12.159132] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.166952] systemd[1]: Detected virtualization kvm. [ 12.168703] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.715386] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.719430] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.724737] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.728723] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.732303] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.742448] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.752624] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System...[ 12.806655] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.264705] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.663552] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.691351] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.858558] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.867935] EDAC sbridge: Ver: 1.1.2 [ 15.815223] Key type dns_resolver registered [ 16.150578] NFS: Registering the id_resolver key type [ 16.156350] Key type id_resolver registered [ 16.158989] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-server login: [ 84.298813] libcfs: loading out-of-tree module taints kernel. [ 84.372885] Key type ._llcrypt registered [ 84.376642] Key type .llcrypt registered [ 84.534873] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 107.549780] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 109.699198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.711735] alg: No test for adler32 (adler32-zlib) [ 110.983419] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 111.491303] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 113.351212] Key type lgssc registered [ 114.817247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.559006] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.816290] hrtimer: interrupt took 5144833 ns [ 179.602606] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 191.886957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.916810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.136291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.188508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.306391] Lustre: lustre-MDT0000: new disk, initializing [ 193.414523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.428905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.218477] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.298417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.387825] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 211.433378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 211.443517] Lustre: Skipped 1 previous similar message [ 211.515818] Lustre: lustre-MDT0001: new disk, initializing [ 211.579163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.617892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.638818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.785035] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.636429] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.379209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.723960] Lustre: lustre-OST0000: new disk, initializing [ 231.730654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.753447] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.851838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.111875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.119552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.231598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.343803] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.792340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.963260] Lustre: lustre-OST0001: new disk, initializing [ 254.966522] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.970432] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.039163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.183661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.196737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.250731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.379253] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.359874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.128356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.966847] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing check_logdir /tmp/testlogs/ [ 294.210513] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing yml_node [ 298.797631] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 301.621724] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 304.055106] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 305.608739] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 17:25:07 EDT 2026 [ 322.013150] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 324.211929] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.938157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 339.944863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.966614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.032438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.034492] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.055460] Lustre: Skipped 2 previous similar messages [ 349.152319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.167954] Lustre: Skipped 6 previous similar messages [ 354.271143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.274655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.288647] Lustre: Skipped 3 previous similar messages [ 354.569047] Lustre: server umount lustre-MDT0000 complete [ 359.324918] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865962 with bad export cookie 10429237051826741505 [ 359.337905] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.342507] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 359.407147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.422817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.431219] Lustre: Skipped 1 previous similar message [ 359.456888] Lustre: Skipped 1 previous similar message [ 364.513205] LustreError: 6521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.531377] LustreError: 6521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 365.875550] Lustre: server umount lustre-MDT0001 complete [ 370.378804] Lustre: server umount lustre-OST0000 complete [ 375.299834] Lustre: server umount lustre-OST0001 complete [ 382.928815] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 393.233704] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 430.544949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.784725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 430.820766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 430.917743] Lustre: lustre-MDT0000: new disk, initializing [ 431.031789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.050123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 435.756134] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 447.131773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.308795] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 447.434117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 447.441842] Lustre: Skipped 1 previous similar message [ 447.533183] Lustre: lustre-MDT0001: new disk, initializing [ 447.623732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.673130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.686296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 452.793348] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 462.162845] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.247436] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.735611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 478.987825] Lustre: lustre-OST0000: new disk, initializing [ 478.990861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 479.000726] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.090396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.138624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 480.145563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 480.211401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.222783] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 496.125183] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 502.051451] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.353483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.727533] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.902720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.946421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.953272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.957205] LustreError: Skipped 1 previous similar message [ 510.961773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.970723] Lustre: Skipped 1 previous similar message [ 510.978124] Lustre: Skipped 3 previous similar messages [ 524.255186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 524.541022] Lustre: server umount lustre-OST0000 complete [ 529.891674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 529.899246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.912464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.914927] Lustre: Skipped 4 previous similar messages [ 534.908513] Lustre: server umount lustre-MDT0000 complete [ 536.546046] LustreError: 15086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.569996] LustreError: 15086:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 538.556609] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866141 with bad export cookie 10429237051826743038 [ 538.561317] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.571605] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.045117] Lustre: server umount lustre-MDT0001 complete [ 544.275896] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 17:29:05 (1787866145) [ 551.582462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.043363] LustreError: 19312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.125474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.626267] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 557.547495] LustreError: 19313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.636834] LustreError: 19312:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.681754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.448720] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 577.220671] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.650813] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.613962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.802582] LustreError: 21132:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.896242] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 604.541218] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 608.155276] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 616.416650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 616.435497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.451850] Lustre: Skipped 1 previous similar message [ 616.457575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.462159] Lustre: Skipped 2 previous similar messages [ 622.339854] Lustre: server umount lustre-MDT0000 complete [ 625.812777] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866228 with bad export cookie 10429237051826743920 [ 625.834564] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 625.843146] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.106704] Lustre: server umount lustre-MDT0001 complete [ 639.855118] Lustre: server umount lustre-OST0000 complete [ 666.150779] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 680.406528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.766419] LustreError: 23697:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.864678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.873540] Lustre: Skipped 2 previous similar messages [ 685.139963] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 694.183542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.701493] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 702.924995] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 709.793995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.297206] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 725.069279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 725.246520] Lustre: lustre-OST0001: new disk, initializing [ 725.250229] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 725.257827] Lustre: 26012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 727.249028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 727.262766] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 727.384477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 731.664226] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 740.187329] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 743.832983] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 756.197710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.203825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.212754] Lustre: Skipped 2 previous similar messages [ 756.216443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.220458] Lustre: Skipped 5 previous similar messages [ 761.794740] Lustre: server umount lustre-MDT0000 complete [ 763.367838] LustreError: 24428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.388501] LustreError: 24428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 765.652617] LustreError: 26011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866368 with bad export cookie 10429237051826744767 [ 765.654289] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.663702] LustreError: 26011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 766.080263] Lustre: server umount lustre-MDT0001 complete [ 780.143058] Lustre: server umount lustre-OST0000 complete [ 793.339584] Lustre: server umount lustre-OST0001 complete [ 808.104385] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 816.870484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.243950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.248769] Lustre: Skipped 3 previous similar messages [ 820.525056] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 828.402364] LustreError: 29032:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.418737] LustreError: 29032:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 829.471974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.100542] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 838.701750] Lustre: 30177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 845.636277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.842385] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 862.184129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 868.034991] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 875.404591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 878.898425] Lustre: 32056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 884.199853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.240727] Lustre: Skipped 5 previous similar messages [ 884.255481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.261367] Lustre: Skipped 7 previous similar messages [ 890.363663] Lustre: server umount lustre-MDT0000 complete [ 893.700567] LustreError: 29012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866496 with bad export cookie 10429237051826746062 [ 893.705404] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.711770] LustreError: 29012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 894.105490] Lustre: server umount lustre-MDT0001 complete [ 908.541443] Lustre: server umount lustre-OST0000 complete [ 910.626976] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866497/real 1787866497] req@ffff9718b55c1a40 x1874713079220480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787866513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 911.995244] Lustre: server umount lustre-OST0001 complete [ 920.852331] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 17:35:22 (1787866522) [ 948.730615] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 958.501936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.785129] LustreError: 34858:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.803723] LustreError: 34858:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 963.405813] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 972.358849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.778723] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 979.601973] Lustre: 36008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 986.016513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.294332] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 999.281061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.782189] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1012.133651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1015.426440] Lustre: 37884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1023.971384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.983797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.991686] Lustre: Skipped 2 previous similar messages [ 1026.434405] Lustre: server umount lustre-MDT0000 complete [ 1029.663857] LustreError: 34839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866632 with bad export cookie 10429237051826747168 [ 1029.666061] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1029.672531] LustreError: 34839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1029.883210] Lustre: server umount lustre-MDT0001 complete [ 1044.162834] Lustre: server umount lustre-OST0000 complete [ 1057.935273] Lustre: server umount lustre-OST0001 complete [ 1065.815471] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 17:37:47 (1787866667) [ 1080.713799] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1089.693620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.061780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.074130] Lustre: Skipped 7 previous similar messages [ 1094.972360] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1104.026682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.327125] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1111.115841] Lustre: 41175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1118.901458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.044802] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1139.780731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1147.788964] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1155.934957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1160.101244] Lustre: 43054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1161.226419] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1161.236975] Lustre: Skipped 1 previous similar message [ 1165.806334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.817344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.820598] Lustre: Skipped 4 previous similar messages [ 1165.831444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1165.834636] Lustre: Skipped 11 previous similar messages [ 1170.566373] Lustre: server umount lustre-MDT0000 complete [ 1173.911160] LustreError: 40007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866776 with bad export cookie 10429237051826748274 [ 1173.914460] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.918311] LustreError: 40007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1188.320330] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1188.547302] Lustre: server umount lustre-MDT0001 complete [ 1197.139033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.367311] LustreError: 43917:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1198.375822] LustreError: 43917:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9718ad4e70c0 x1874713079318272/t0(0) o250->MGC192.168.201.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787866801 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1198.406954] LustreError: 43917:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1203.633426] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1212.086691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.027372] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1218.555479] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1224.126415] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1225.604558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1230.686248] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1232.307793] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1237.984592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1243.467391] Lustre: server umount lustre-MDT0000 complete [ 1244.147327] LustreError: 43929:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.163539] LustreError: 43929:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [ 1247.891525] LustreError: 40008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866850 with bad export cookie 10429237051826749373 [ 1247.905355] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.909214] LustreError: 40008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1248.520217] Lustre: server umount lustre-MDT0001 complete [ 1261.466702] Lustre: server umount lustre-OST0000 complete [ 1274.373866] Lustre: server umount lustre-OST0001 complete [ 1282.460189] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 17:41:24 (1787866884) [ 1297.724735] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1309.594452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.233289] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1324.724732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.159942] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1333.065536] Lustre: 48818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1339.710408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1347.193298] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1356.600848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.137477] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1375.660222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1380.815932] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1393.998962] Lustre: server umount lustre-MDT0000 complete [ 1397.687143] LustreError: 47646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867000 with bad export cookie 10429237051826750192 [ 1397.696896] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1397.709302] LustreError: 47646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1398.238522] Lustre: server umount lustre-MDT0001 complete [ 1412.353238] Lustre: server umount lustre-OST0000 complete [ 1414.111120] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787867001/real 1787867001] req@ffff971886568000 x1874713079410816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787867017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1416.658023] Lustre: server umount lustre-OST0001 complete [ 1426.732254] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 17:43:48 (1787867028) [ 1429.637455] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1431.303187] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 17:43:52 (1787867032) [ 1443.451374] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 1452.482935] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1474.510301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.779675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1474.857351] Lustre: lustre-MDT0000: new disk, initializing [ 1474.958896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1480.291395] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1489.350877] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.857923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1496.174743] Lustre: lustre-OST0000: new disk, initializing [ 1496.177821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1496.183306] Lustre: Skipped 1 previous similar message [ 1496.188603] Lustre: 54893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1502.760020] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1503.222659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1503.234413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1503.269453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1512.548211] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1514.843844] Lustre: DEBUG MARKER: create test files [ 1548.771535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.797757] Lustre: Skipped 13 previous similar messages [ 1551.731329] Lustre: server umount lustre-OST0000 complete [ 1555.787277] Lustre: server umount lustre-MDT0000 complete [ 1567.954066] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1570.952688] Key type lgssc unregistered [ 1571.155685] LNet: 56526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.171496] LNetError: 56526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.200381] LNet: Removed LNI 192.168.201.121@tcp [ 1573.304403] Key type .llcrypt unregistered [ 1573.307771] Key type ._llcrypt unregistered [ 1601.123864] Key type ._llcrypt registered [ 1601.126521] Key type .llcrypt registered [ 1601.303995] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1602.381809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1602.591265] alg: No test for adler32 (adler32-zlib) [ 1603.686294] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1603.921083] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1605.623876] Key type lgssc registered [ 1606.770710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.553883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.588748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.145868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.357555] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1629.641695] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1637.479394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.963647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.054580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1642.267797] Lustre: 59261:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1642.276866] Lustre: 59261:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1642.288452] Lustre: 59261:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1642.303835] Lustre: 59261:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1642.313475] Lustre: 59261:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1642.328394] Lustre: 59261:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1644.824525] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1651.547910] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.589568] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1662.116306] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1698.273180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.286135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.296616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.955626] Lustre: server umount lustre-OST0000 complete [ 1705.180564] Lustre: server umount lustre-MDT0000 complete [ 1716.365826] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1719.822429] Key type lgssc unregistered [ 1720.162127] LNet: 60900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.175699] LNetError: 60900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.202334] LNet: Removed LNI 192.168.201.121@tcp [ 1721.298729] Key type .llcrypt unregistered [ 1721.300162] Key type ._llcrypt unregistered [ 1756.273692] Key type ._llcrypt registered [ 1756.275649] Key type .llcrypt registered [ 1756.458485] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1757.612750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.630874] alg: No test for adler32 (adler32-zlib) [ 1758.620310] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1758.858759] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1760.647206] Key type lgssc registered [ 1762.213714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1771.189264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1771.236922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.681607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.462554] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1785.290958] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1792.024083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1792.329139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.501192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1797.618699] Lustre: 63737:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 1797.630466] Lustre: 63737:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/1, destroy: 0/0/0 [ 1797.647107] Lustre: 63737:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1797.663312] Lustre: 63737:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1797.674291] Lustre: 63737:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1797.687202] Lustre: 63737:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1798.446880] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1805.507040] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.301468] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1823.201239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.207953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.222813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.687544] Lustre: server umount lustre-OST0000 complete [ 1830.805185] Lustre: server umount lustre-MDT0000 complete [ 1842.008982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1845.548790] Key type lgssc unregistered [ 1845.960104] LNet: 65300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.966142] LNetError: 65300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.991182] LNet: Removed LNI 192.168.201.121@tcp [ 1846.945609] Key type .llcrypt unregistered [ 1846.947870] Key type ._llcrypt unregistered [ 1868.529907] Key type ._llcrypt registered [ 1868.532982] Key type .llcrypt registered [ 1868.695285] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 1885.814394] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1887.643766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1887.829059] alg: No test for adler32 (adler32-zlib) [ 1889.118521] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1889.564325] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1891.311177] Key type lgssc registered [ 1892.402740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.927343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1928.956864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.315693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1930.374670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1930.498779] Lustre: lustre-MDT0000: new disk, initializing [ 1930.694434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.730540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1935.331288] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1946.343445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.548070] Lustre: 68453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1946.611854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1946.616279] Lustre: Skipped 1 previous similar message [ 1946.676067] Lustre: lustre-MDT0001: new disk, initializing [ 1946.815961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.880551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1946.892694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1951.451971] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1960.380622] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1967.637812] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1974.783619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1975.331824] Lustre: lustre-OST0000: new disk, initializing [ 1975.340958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1975.349870] Lustre: 70423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1975.488200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.589410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1982.607867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1982.656953] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1982.766795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1990.569040] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1994.639844] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1994.918368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1998.849209] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1998.984273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2003.425531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.429469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.455404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.475562] Lustre: Skipped 1 previous similar message [ 2008.546477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.552825] Lustre: Skipped 1 previous similar message [ 2013.672053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.679399] Lustre: Skipped 1 previous similar message [ 2015.199107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2015.373506] Lustre: server umount lustre-OST0000 complete [ 2023.905150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.909729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.918135] Lustre: Skipped 2 previous similar messages [ 2023.935691] Lustre: Skipped 1 previous similar message [ 2025.966641] Lustre: server umount lustre-MDT0000 complete [ 2029.996357] LustreError: 68446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867632 with bad export cookie 13509348780388115154 [ 2030.007732] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2030.013286] LustreError: 68446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2030.058196] LustreError: 66678:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 2030.365643] Lustre: server umount lustre-MDT0001 complete [ 2038.863211] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 17:54:00 (1787867640) [ 2049.498464] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2091.222646] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2102.452377] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2141.992739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.347764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2142.371691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2142.461711] Lustre: lustre-MDT0000: new disk, initializing [ 2142.561434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.584830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2146.480817] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2158.913085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.072593] Lustre: 76442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2159.141542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2159.150340] Lustre: Skipped 1 previous similar message [ 2159.217221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2159.232511] Lustre: lustre-MDT0001: new disk, initializing [ 2159.351618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2159.412983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2159.438535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2164.389776] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2174.908098] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.232331] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2186.811419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.251266] Lustre: lustre-OST0000: new disk, initializing [ 2187.255412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2187.267637] Lustre: 78415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2187.480524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.824344] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2196.081648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2196.099367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2196.293029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2204.024600] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2209.127413] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2209.333584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2213.884565] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2214.052370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2216.932693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.943293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.957709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.964520] Lustre: Skipped 1 previous similar message [ 2230.240397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2230.478995] Lustre: server umount lustre-OST0000 complete [ 2236.384956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2236.396250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.429134] Lustre: Skipped 1 previous similar message [ 2236.440178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.453797] Lustre: Skipped 4 previous similar messages [ 2240.784228] Lustre: server umount lustre-MDT0000 complete [ 2242.528552] LustreError: 79757:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.547096] LustreError: 79757:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2245.727040] LustreError: 76437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867848 with bad export cookie 13509348780388116036 [ 2245.751676] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2246.423604] Lustre: server umount lustre-MDT0001 complete [ 2253.954821] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 17:57:35 (1787867855) [ 2259.495160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.805047] LustreError: 80831:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.862823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.013422] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2265.056230] LustreError: 80832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.153791] LustreError: 80831:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.403703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.784537] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2286.247942] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.752451] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.119867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.462912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.467077] Lustre: Skipped 1 previous similar message [ 2304.001110] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2311.669776] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2315.051884] Lustre: 83502:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 04591740-b08a-41a0-a539-4718bf3c70cb at adminstrative request [ 2315.242959] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.250921] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2325.500619] Lustre: MGS: Client 04591740-b08a-41a0-a539-4718bf3c70cb (at 0@lo) reconnecting [ 2325.513479] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xbb7ad923e1e3e9af to 0xbb7ad923e1e3ec4f [ 2325.533794] Lustre: MGC192.168.201.121@tcp: Connection restored to 0@lo (at 0@lo) [ 2353.231642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.461555] Lustre: lustre-OST0001: new disk, initializing [ 2353.468122] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2353.474329] Lustre: 83896:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2359.438205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2359.450968] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2359.642888] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2359.898765] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2368.672123] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2375.141252] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.149873] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.153489] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2375.167068] Lustre: Skipped 2 previous similar messages [ 2375.188269] Lustre: Skipped 3 previous similar messages [ 2377.335307] Lustre: server umount lustre-OST0001 complete [ 2380.263886] LustreError: 82654:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.290908] LustreError: 82654:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2381.797787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2381.808184] LustreError: Skipped 1 previous similar message [ 2383.844865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.860192] Lustre: Skipped 1 previous similar message [ 2386.920631] Lustre: server umount lustre-OST0000 complete [ 2390.502539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2390.513624] LustreError: Skipped 1 previous similar message [ 2396.475569] Lustre: server umount lustre-MDT0000 complete [ 2399.688294] LustreError: 80811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868002 with bad export cookie 13509348780388117583 [ 2399.690783] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.699568] LustreError: 80811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2400.030489] Lustre: server umount lustre-MDT0001 complete [ 2407.696644] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 18:00:09 (1787868009) [ 2409.426304] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2410.963577] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 18:00:12 (1787868012) [ 2412.596208] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2414.334208] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 18:00:15 (1787868015) [ 2415.712705] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2417.406290] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 18:00:18 (1787868018) [ 2418.836273] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2420.416449] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 18:00:21 (1787868021) [ 2421.785702] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2427.481070] loop: module loaded [ 2427.492486] loop0: detected capacity change from 0 to 4294967296 [ 2427.543679] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2429.356943] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2432.986290] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2436.650686] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2456.718742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.788828] LustreError: 88019:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2456.797685] LustreError: 88019:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2456.801719] LustreError: 88019:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2456.810470] LustreError: 88019:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2486.842223] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2498.402804] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2536.519035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.787398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2536.823093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2536.951330] Lustre: lustre-MDT0000: new disk, initializing [ 2537.012531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.021253] Lustre: Skipped 1 previous similar message [ 2537.043931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2541.190910] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2551.109576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.213938] Lustre: 90912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2551.596740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2556.231878] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2565.937455] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2571.605785] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2577.891773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2578.150566] Lustre: 92880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2583.833859] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2585.661029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2585.759914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2592.449072] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2599.091174] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2599.379592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2605.880994] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2606.197118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2611.681894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2611.690562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.705981] Lustre: Skipped 3 previous similar messages [ 2611.707103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.723836] Lustre: Skipped 9 previous similar messages [ 2614.625586] Lustre: server umount lustre-OST0000 complete [ 2625.055506] Lustre: server umount lustre-MDT0000 complete [ 2627.042806] LustreError: 90922:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.069802] LustreError: 90922:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2628.533767] LustreError: 90903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868231 with bad export cookie 13509348780388118199 [ 2628.534793] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.546224] LustreError: 90903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2628.764634] Lustre: server umount lustre-MDT0001 complete [ 2637.043391] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 18:03:58 (1787868238) [ 2644.822620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.490679] LustreError: 95294:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.940676] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2667.532478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.055900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2668.072886] Lustre: Skipped 3 previous similar messages [ 2674.219186] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2682.956586] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2690.023415] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2697.224861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2703.551069] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2713.567516] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.343545] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2721.949299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2729.077979] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2729.367553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2738.299813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2738.512809] Lustre: lustre-OST0001: new disk, initializing [ 2738.516097] Lustre: Skipped 2 previous similar messages [ 2738.519664] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2738.524260] Lustre: Skipped 3 previous similar messages [ 2738.528492] Lustre: 98668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2745.964262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2745.978165] Lustre: Skipped 2 previous similar messages [ 2745.987286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2746.177908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2746.479993] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2756.894390] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2762.463770] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2762.837808] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2766.873843] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2767.075220] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2787.371723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2811.709766] Lustre: Failing over lustre-MDT0000 [ 2812.072280] Lustre: server umount lustre-MDT0000 complete [ 2812.903501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.908288] LustreError: 95295:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.919319] Lustre: Skipped 4 previous similar messages [ 2812.941769] LustreError: 95295:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2828.255252] Lustre: 66675:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868415/real 1787868415] req@00000000fe8f324c x1874714943570944/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787868431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2828.299072] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.084877] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2836.086930] LDISKFS-fs (dm-0): recovery complete [ 2836.094528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.175275] LustreError: 101483:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2838.185462] LustreError: 101483:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000057d1aee x1874714943576448/t0(0) o250->MGC192.168.201.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787868441 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2838.207054] LustreError: 101483:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2838.517728] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbb7ad923e1e4b6ca [ 2838.531709] Lustre: MGC192.168.201.121@tcp: Connection restored to 0@lo (at 0@lo) [ 2838.868814] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2841.982676] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2841.984243] Lustre: 99198:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2843.609561] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2844.129110] Lustre: 95295:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2844.142479] Lustre: 95295:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 1 previous similar message [ 2844.166976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2844.278715] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2844.608075] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2844.950080] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2844.972044] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 1 previous similar message [ 2845.295226] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2845.302970] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 1 previous similar message [ 2845.327042] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2845.336544] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 7 previous similar messages [ 2845.967981] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2845.983186] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 2 previous similar messages [ 2846.304780] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2846.308022] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 2 previous similar messages [ 2847.615374] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2847.646199] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 6 previous similar messages [ 2847.969439] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2847.988180] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 5 previous similar messages [ 2848.312592] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2848.318755] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 5 previous similar messages [ 2851.810220] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2851.823064] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 12 previous similar messages [ 2852.169872] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2852.198907] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 12 previous similar messages [ 2852.519138] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2852.534362] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 12 previous similar messages [ 2859.821503] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2859.833185] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 24 previous similar messages [ 2860.473373] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2860.493634] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 25 previous similar messages [ 2860.834592] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2860.844234] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 25 previous similar messages [ 2876.080288] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2876.103958] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 50 previous similar messages [ 2876.752516] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2876.762443] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 50 previous similar messages [ 2877.095918] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2877.113721] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 50 previous similar messages [ 2901.311196] Lustre: 101515:0:(ldlm_lib.c:2457:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2901.331054] Lustre: 101515:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2901.352282] Lustre: 101515:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client eeef5ecc-2e90-4838-a8c2-22d019bd959d@192.168.201.21@tcp [ 2901.379848] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2901.392584] Lustre: 101515:0:(ldlm_lib.c:1977:abort_req_replay_queue()) @@@ aborted: req@000000003f4d95d0 x1874714925535744/t0(8589934775) o101->eeef5ecc-2e90-4838-a8c2-22d019bd959d@192.168.201.21@tcp:579/0 lens 592/0 e 0 to 0 dl 1787868514 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2901.431225] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2901.445243] Lustre: lustre-MDT0000: Denying connection for new client eeef5ecc-2e90-4838-a8c2-22d019bd959d (at 192.168.201.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2901.457427] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2901.521727] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2901.526674] Lustre: Skipped 2 previous similar messages [ 2901.536258] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2901.578354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2901.578595] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 2915.814280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.844168] Lustre: Skipped 1 previous similar message [ 2915.859493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2915.872369] Lustre: Skipped 3 previous similar messages [ 2920.729823] Lustre: server umount lustre-OST0000 complete [ 2920.931857] LustreError: 97840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.981115] LustreError: 97840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 35 previous similar messages [ 2925.536893] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2925.545165] LustreError: Skipped 1 previous similar message [ 2931.025614] Lustre: server umount lustre-OST0001 complete [ 2941.320604] Lustre: server umount lustre-MDT0000 complete [ 2945.401781] LustreError: 98666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868548 with bad export cookie 13509348780388169418 [ 2945.418065] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2945.421292] LustreError: 98666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2945.774781] Lustre: server umount lustre-MDT0001 complete [ 2954.265574] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 18:09:15 (1787868555) [ 2962.176188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.184378] Lustre: *** cfs_fail_loc=197, val=0*** [ 2977.826279] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2982.943480] LustreError: 103493:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2988.159149] Lustre: *** cfs_fail_loc=197, val=0*** [ 2988.161641] Lustre: Skipped 65 previous similar messages [ 3008.735264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.741693] Lustre: Skipped 3 previous similar messages [ 3014.620194] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3023.846180] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3026.403468] Lustre: server umount lustre-OST0000 complete [ 3036.046527] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 18:10:37 (1787868637) [ 3058.512428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.049161] LustreError: 105117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.074077] LustreError: 105117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3064.110376] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3074.110906] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3078.042235] LustreError: 105148:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3078.055515] LustreError: 105148:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3078.063929] LustreError: 105148:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3078.556034] Lustre: server umount lustre-MDT0000 complete [ 3112.584313] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3123.747920] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3161.063356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.363513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3161.408654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3161.567493] Lustre: lustre-MDT0000: new disk, initializing [ 3161.758658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3166.398982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3180.484867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.732671] Lustre: 109320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3181.289053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3187.973583] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3197.619712] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3204.189363] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3210.785908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.981243] Lustre: 111291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3217.096721] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3218.108696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3227.044166] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3232.156298] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3232.422383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3237.654402] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3237.843214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3244.027326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.039909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.053897] Lustre: Skipped 5 previous similar messages [ 3244.057794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.068174] LustreError: Skipped 1 previous similar message [ 3244.098794] Lustre: Skipped 6 previous similar messages [ 3254.240040] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3254.535651] Lustre: server umount lustre-OST0000 complete [ 3264.987996] Lustre: server umount lustre-MDT0000 complete [ 3269.781876] LustreError: 109312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868872 with bad export cookie 13509348780388180058 [ 3269.785435] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3269.800065] LustreError: 109312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3270.366821] Lustre: server umount lustre-MDT0001 complete [ 3281.616763] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 18:14:43 (1787868883) [ 3291.779849] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3294.507986] Key type lgssc unregistered [ 3294.856177] LNet: 113784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3294.874474] LNetError: 113784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3295.911983] LNet: Removed LNI 192.168.201.121@tcp [ 3296.961567] Key type .llcrypt unregistered [ 3296.963739] Key type ._llcrypt unregistered [ 3317.700570] Key type ._llcrypt registered [ 3317.704597] Key type .llcrypt registered [ 3317.820501] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3336.115328] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3338.039784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3338.094183] alg: No test for adler32 (adler32-zlib) [ 3339.233759] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3339.541913] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3341.287168] Key type lgssc registered [ 3342.999764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3385.965235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3385.990853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.250258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3387.328215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3387.500783] Lustre: lustre-MDT0000: new disk, initializing [ 3387.590756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.632797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3393.119633] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3405.575196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.879680] Lustre: lustre-OST0000: new disk, initializing [ 3405.888776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3405.901903] Lustre: Skipped 1 previous similar message [ 3405.909272] Lustre: 117961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3405.999682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.633033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3411.650634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3411.691869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3412.989790] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3423.693880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3423.861684] Lustre: lustre-OST0001: new disk, initializing [ 3423.865136] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3423.869888] Lustre: 118824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3423.969347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3429.818337] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3431.616065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3431.622781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3431.673089] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3441.634068] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.648332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3441.653171] Lustre: Skipped 1 previous similar message [ 3444.902976] Lustre: server umount lustre-MDT0000 complete [ 3451.818804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3456.431446] LustreError: 117149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869059 with bad export cookie 16908075525739252544 [ 3456.445127] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3456.459570] LustreError: 117149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3456.526799] Lustre: server umount lustre-OST0000 complete [ 3460.146683] Lustre: server umount lustre-OST0001 complete [ 3469.929903] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 18:17:50 (1787869070) [ 3494.200955] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 18:18:15 (1787869095) [ 3506.120079] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3530.272253] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 18:18:51 (1787869131) [ 3572.878836] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3585.300359] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3621.836046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.275431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3622.317268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3622.402280] Lustre: lustre-MDT0000: new disk, initializing [ 3622.465156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.489156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3627.442485] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3638.776798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.865810] Lustre: 126578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3638.914844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3638.918634] Lustre: Skipped 1 previous similar message [ 3638.974255] Lustre: lustre-MDT0001: new disk, initializing [ 3639.118878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3639.149966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3639.163668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3644.280773] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3653.889061] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3659.757494] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3666.793600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.059374] Lustre: lustre-OST0000: new disk, initializing [ 3667.062579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3667.067231] Lustre: 128545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3667.144837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.193835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3673.205553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3673.356211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3673.935352] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3683.506980] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3688.525982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3688.771257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3693.860466] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3694.080315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3699.168481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.183552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.194399] Lustre: Skipped 1 previous similar message [ 3699.200331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3699.206181] Lustre: Skipped 1 previous similar message [ 3702.563606] Lustre: server umount lustre-OST0000 complete [ 3709.410329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.413927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3709.437165] Lustre: Skipped 2 previous similar messages [ 3712.897758] Lustre: server umount lustre-MDT0000 complete [ 3715.947732] LustreError: 128556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869318 with bad export cookie 16908075525739253769 [ 3715.956293] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.959909] LustreError: 128556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3716.342094] Lustre: server umount lustre-MDT0001 complete [ 3724.490858] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 18:22:05 (1787869325) [ 3731.392190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.787445] LustreError: 130959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.808974] LustreError: 130959:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3731.892088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.070946] LustreError: 130960:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.378385] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3741.152348] LustreError: 130959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.275137] LustreError: 130960:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.972889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.487451] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3759.160636] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3765.245698] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3772.082334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3772.475934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.489098] Lustre: Skipped 1 previous similar message [ 3779.363449] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3786.856758] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3791.008149] Lustre: *** cfs_fail_loc=159, val=0*** [ 3806.627237] Lustre: lustre-MDT0000: Client 0ac7f7dc-b239-477a-890e-2f79e0632bf3 (at 192.168.201.21@tcp) reconnecting [ 3808.099832] Lustre: *** cfs_fail_loc=159, val=0*** [ 3808.105246] Lustre: Skipped 5 previous similar messages [ 3823.532567] Lustre: lustre-MDT0000: Client 0ac7f7dc-b239-477a-890e-2f79e0632bf3 (at 192.168.201.21@tcp) reconnecting [ 3829.729738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.739025] LustreError: Skipped 1 previous similar message [ 3829.752162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.786929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.789627] Lustre: Skipped 1 previous similar message [ 3830.704638] Lustre: server umount lustre-OST0000 complete [ 3837.921162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.933542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3837.938893] Lustre: Skipped 1 previous similar message [ 3845.387571] Lustre: server umount lustre-MDT0000 complete [ 3847.137187] LustreError: 133622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.174967] LustreError: 133622:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3849.444498] LustreError: 130942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869452 with bad export cookie 16908075525739254644 [ 3849.462371] LustreError: 130942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3849.464962] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.849216] Lustre: server umount lustre-MDT0001 complete [ 3862.646772] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3865.855236] Key type lgssc unregistered [ 3866.079404] LNet: 134736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.086427] LNetError: 134736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3866.102356] LNet: Removed LNI 192.168.201.121@tcp [ 3867.125604] Key type .llcrypt unregistered [ 3867.129705] Key type ._llcrypt unregistered [ 3891.421513] Key type ._llcrypt registered [ 3891.424866] Key type .llcrypt registered [ 3891.550621] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3892.872926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3892.918760] alg: No test for adler32 (adler32-zlib) [ 3894.054802] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3894.413405] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3896.201848] Key type lgssc registered [ 3897.652384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3907.001984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3907.023390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.659560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.926984] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3918.830540] LustreError: 136151:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.844739] LustreError: 136151:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3922.655792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.136655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3928.494802] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3937.201901] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3943.676207] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3952.267618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3952.688488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.726505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3959.406568] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3967.437736] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3977.338322] Lustre: *** cfs_fail_loc=159, val=0*** [ 3994.025560] Lustre: lustre-MDT0000: Client c79697c5-e701-4b79-99cf-84617a6eedd3 (at 192.168.201.21@tcp) reconnecting [ 3996.586122] Lustre: *** cfs_fail_loc=159, val=0*** [ 3996.598577] Lustre: Skipped 2 previous similar messages [ 4012.991969] Lustre: lustre-MDT0001: Client c79697c5-e701-4b79-99cf-84617a6eedd3 (at 192.168.201.21@tcp) reconnecting [ 4014.442893] Lustre: *** cfs_fail_loc=159, val=0*** [ 4014.451285] Lustre: Skipped 1 previous similar message [ 4030.915505] Lustre: lustre-MDT0001: Client c79697c5-e701-4b79-99cf-84617a6eedd3 (at 192.168.201.21@tcp) reconnecting [ 4034.841232] Lustre: 139187:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4037.257604] Lustre: *** cfs_fail_loc=159, val=0*** [ 4037.259411] Lustre: Skipped 4 previous similar messages [ 4053.898213] Lustre: lustre-MDT0000: Client 3eb8b22e-0360-47d6-aff1-eee24b29135d (at 192.168.201.21@tcp) reconnecting [ 4055.688069] Lustre: *** cfs_fail_loc=159, val=0*** [ 4055.691791] Lustre: Skipped 14 previous similar messages [ 4071.400450] Lustre: lustre-MDT0000: Client 3eb8b22e-0360-47d6-aff1-eee24b29135d (at 192.168.201.21@tcp) reconnecting [ 4072.902488] Lustre: 139489:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4079.083240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4079.087642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.121097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.609762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.614050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.187826] Lustre: server umount lustre-OST0000 complete [ 4090.850751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.867408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.881649] Lustre: Skipped 2 previous similar messages [ 4093.980519] Lustre: server umount lustre-MDT0000 complete [ 4095.970867] LustreError: 136152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4097.317316] LustreError: 139309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869700 with bad export cookie 1057403184735855796 [ 4097.334431] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.338982] LustreError: 139309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4097.833943] Lustre: server umount lustre-MDT0001 complete [ 4107.966319] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4111.877964] Key type lgssc unregistered [ 4112.344858] LNet: 140546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4112.356961] LNetError: 140546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4112.377303] LNet: Removed LNI 192.168.201.121@tcp [ 4113.435427] Key type .llcrypt unregistered [ 4113.438057] Key type ._llcrypt unregistered [ 4137.660146] Key type ._llcrypt registered [ 4137.666896] Key type .llcrypt registered [ 4137.772635] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4139.082557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4139.142273] alg: No test for adler32 (adler32-zlib) [ 4140.190423] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4140.494222] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4142.280872] Key type lgssc registered [ 4143.469256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4150.072525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4150.090328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.542086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.346461] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4162.026606] LustreError: 141964:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.044262] LustreError: 141964:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4162.798246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.180728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.686266] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4174.320483] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4179.442527] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4185.523232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.988237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.078948] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4192.688212] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4196.230466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4200.917790] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4207.124532] Lustre: 144545:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4210.306873] Lustre: 144593:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4211.910559] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4212.736291] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4215.625553] Lustre: 144740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4216.285220] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4217.220912] Lustre: 144838:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4221.922913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.932363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.935665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4226.937807] Lustre: server umount lustre-OST0000 complete [ 4232.162873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.184583] Lustre: Skipped 1 previous similar message [ 4232.202329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.205026] Lustre: Skipped 2 previous similar messages [ 4236.850487] Lustre: server umount lustre-MDT0000 complete [ 4237.285938] LustreError: 141965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.308418] LustreError: 141965:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4240.765370] LustreError: 141944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869843 with bad export cookie 12509673676213171376 [ 4240.809881] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.500550] Lustre: server umount lustre-MDT0001 complete [ 4252.162286] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4255.449339] Key type lgssc unregistered [ 4255.964625] LNet: 145904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4255.977935] LNetError: 145904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4255.998677] LNet: Removed LNI 192.168.201.121@tcp [ 4257.059135] Key type .llcrypt unregistered [ 4257.060270] Key type ._llcrypt unregistered [ 4281.430948] Key type ._llcrypt registered [ 4281.435497] Key type .llcrypt registered [ 4281.549406] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4282.834335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4282.892183] alg: No test for adler32 (adler32-zlib) [ 4284.006179] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4284.239428] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4285.951582] Key type lgssc registered [ 4287.361455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.960312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4294.977616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.371921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.494372] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4306.914669] LustreError: 147318:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.925989] LustreError: 147318:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4307.822470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.157344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.443746] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4320.036918] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4326.135189] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4334.067706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4334.605228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.752938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4343.115143] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4351.898886] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4353.007940] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4357.717171] Lustre: *** cfs_fail_loc=159, val=0*** [ 4373.424321] Lustre: lustre-MDT0000: Client 609887d7-af14-4737-ac27-2674161141a4 (at 192.168.201.21@tcp) reconnecting [ 4392.931770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4392.945830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.961527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4393.964761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4393.965923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4393.966566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.995444] Lustre: Skipped 1 previous similar message [ 4397.424372] Lustre: server umount lustre-OST0000 complete [ 4404.195401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.207642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4404.216502] Lustre: Skipped 1 previous similar message [ 4408.110843] Lustre: server umount lustre-MDT0000 complete [ 4409.312382] LustreError: 147314:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.326378] LustreError: 147314:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4412.070737] LustreError: 149169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870014 with bad export cookie 5073384120652457649 [ 4412.085174] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4412.097827] LustreError: 149169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4412.368511] Lustre: server umount lustre-MDT0001 complete [ 4424.088446] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4426.824825] Key type lgssc unregistered [ 4427.156995] LNet: 150989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4427.173502] LNetError: 150989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4427.194888] LNet: Removed LNI 192.168.201.121@tcp [ 4428.063753] Key type .llcrypt unregistered [ 4428.066030] Key type ._llcrypt unregistered [ 4453.493650] Key type ._llcrypt registered [ 4453.497429] Key type .llcrypt registered [ 4453.766046] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4455.249365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4455.430657] alg: No test for adler32 (adler32-zlib) [ 4456.601623] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4456.946815] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4458.713042] Key type lgssc registered [ 4460.060725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4469.439777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4469.469931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.042216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.737091] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4480.490683] LustreError: 152403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.511755] LustreError: 152403:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4485.606270] LustreError: 152404:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.328682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.915036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4493.064529] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4501.275249] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4508.374996] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4517.780980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.389686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.194666] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4531.718471] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4532.647184] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4540.935110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4563.411907] Lustre: lustre-MDT0001: evicting a470b7b5-16e7-42ef-a483-f13ef9af818c (at 192.168.201.21@tcp) by administrative request [ 4563.419086] Lustre: Skipped 1 previous similar message [ 4592.100470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.101567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4592.124463] Lustre: Skipped 1 previous similar message [ 4592.125563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.241093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.257525] Lustre: Skipped 2 previous similar messages [ 4598.454722] Lustre: server umount lustre-OST0000 complete [ 4604.386687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4604.401906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.418333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4607.458753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4607.460244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.580111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4612.589496] Lustre: Skipped 2 previous similar messages [ 4618.207809] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4618.563453] Lustre: server umount lustre-MDT0000 complete [ 4622.492663] LustreError: 152385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870225 with bad export cookie 9726648399965818258 [ 4622.501176] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4622.510846] LustreError: 152385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4622.905313] Lustre: server umount lustre-MDT0001 complete [ 4634.483613] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4637.921900] Key type lgssc unregistered [ 4638.187921] LNet: 156501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4638.199552] LNetError: 156501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4638.212669] LNet: Removed LNI 192.168.201.121@tcp [ 4639.446967] Key type .llcrypt unregistered [ 4639.449148] Key type ._llcrypt unregistered [ 4719.175529] Key type ._llcrypt registered [ 4719.179495] Key type .llcrypt registered [ 4719.333140] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4720.341049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4720.440226] alg: No test for adler32 (adler32-zlib) [ 4721.444663] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4721.592931] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4723.271502] Key type lgssc registered [ 4724.099324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4732.374253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4732.394060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.053328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.995591] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4743.149925] LustreError: 158995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.170515] LustreError: 158995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4748.010383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.272552] LustreError: 158996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.711595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4753.809501] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4764.652802] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4770.330917] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4778.014185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4778.371333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.457293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4785.242857] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4792.784615] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4793.780270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4804.066874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.077461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4804.087432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4804.088519] Lustre: Skipped 1 previous similar message [ 4806.718892] Lustre: server umount lustre-OST0000 complete [ 4814.307093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.310180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.329825] Lustre: Skipped 1 previous similar message [ 4814.349335] Lustre: Skipped 2 previous similar messages [ 4817.015596] Lustre: server umount lustre-MDT0000 complete [ 4819.428890] LustreError: 158995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.453516] LustreError: 158995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4820.576912] LustreError: 158976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870423 with bad export cookie 3275888503623865365 [ 4820.586977] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.595057] LustreError: 158976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4820.842641] Lustre: server umount lustre-MDT0001 complete [ 4834.401547] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4837.081306] Key type lgssc unregistered [ 4837.412219] LNet: 162540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4837.419665] LNetError: 162540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4837.433148] LNet: Removed LNI 192.168.201.121@tcp [ 4838.277546] Key type .llcrypt unregistered [ 4838.279097] Key type ._llcrypt unregistered [ 4863.372217] Key type ._llcrypt registered [ 4863.373661] Key type .llcrypt registered [ 4863.504161] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4864.717648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4864.768050] alg: No test for adler32 (adler32-zlib) [ 4866.059873] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 4866.380176] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4868.127193] Key type lgssc registered [ 4869.463828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4886.947056] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 18:41:28 (1787870488) [ 4906.011818] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4917.131339] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4953.523207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4953.553338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.968145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4955.014120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4955.097543] Lustre: lustre-MDT0000: new disk, initializing [ 4955.180675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.195265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4959.915695] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4970.915197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.036971] Lustre: 166805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4971.076891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4971.080555] Lustre: Skipped 1 previous similar message [ 4971.148690] Lustre: lustre-MDT0001: new disk, initializing [ 4971.240871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4971.291925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4971.314644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4976.260300] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4985.469978] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4991.259799] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4999.004876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4999.290544] Lustre: lustre-OST0000: new disk, initializing [ 4999.295724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4999.303294] Lustre: 168773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4999.420074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.978064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5008.984030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5009.057243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5009.284613] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5021.235422] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5034.976077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5034.979874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.984463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5034.985609] Lustre: Skipped 1 previous similar message [ 5038.425622] Lustre: server umount lustre-OST0000 complete [ 5043.687101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5043.702140] LustreError: Skipped 1 previous similar message [ 5043.711700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.735524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5043.740495] Lustre: Skipped 1 previous similar message [ 5045.216207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.217741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5045.252776] Lustre: Skipped 1 previous similar message [ 5049.332128] Lustre: server umount lustre-MDT0000 complete [ 5053.508090] LustreError: 166797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870656 with bad export cookie 10151217496625022877 [ 5053.511718] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5053.520281] LustreError: 166797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5054.182496] Lustre: server umount lustre-MDT0001 complete [ 5065.774516] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5068.984491] Key type lgssc unregistered [ 5069.422597] LNet: 170684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5069.434187] LNetError: 170684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5069.468878] LNet: Removed LNI 192.168.201.121@tcp [ 5070.289593] Key type .llcrypt unregistered [ 5070.293957] Key type ._llcrypt unregistered [ 5099.710397] Key type ._llcrypt registered [ 5099.712322] Key type .llcrypt registered [ 5099.848506] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5100.929470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5100.957131] alg: No test for adler32 (adler32-zlib) [ 5102.131842] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5102.451235] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5104.159204] Key type lgssc registered [ 5105.282643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5114.133911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5114.151871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.611476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.055381] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5126.121432] LustreError: 172102:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.141291] LustreError: 172102:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5130.009453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.655304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5136.128963] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5143.574922] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5150.160173] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5157.119369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5157.520846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.657261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5164.194646] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5171.972168] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5181.286267] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5181.306286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.320748] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5181.327572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.348898] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5182.282811] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5182.304642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.312318] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5182.317694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.329054] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5183.241220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.055878] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5184.060182] Lustre: Skipped 1 previous similar message [ 5184.085455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.093435] Lustre: Skipped 1 previous similar message [ 5184.097974] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5184.102853] Lustre: Skipped 1 previous similar message [ 5184.106833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.118894] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5184.125970] Lustre: Skipped 1 previous similar message [ 5185.163903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.094491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5186.098718] Lustre: Skipped 1 previous similar message [ 5186.112346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.119965] Lustre: Skipped 1 previous similar message [ 5186.124264] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5186.128471] Lustre: Skipped 1 previous similar message [ 5186.133692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.155914] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5186.177107] Lustre: Skipped 1 previous similar message [ 5186.990442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.176283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.344313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.588590] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5190.595243] Lustre: Skipped 3 previous similar messages [ 5190.617521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.627612] Lustre: Skipped 3 previous similar messages [ 5190.633458] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5190.635905] Lustre: Skipped 3 previous similar messages [ 5190.643037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.655297] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5190.657764] Lustre: Skipped 3 previous similar messages [ 5191.760487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.765931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.848777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.037878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.980447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.952957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.846848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.836481] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5198.841412] Lustre: Skipped 7 previous similar messages [ 5198.857141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.869678] Lustre: Skipped 7 previous similar messages [ 5198.872921] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5198.877423] Lustre: Skipped 7 previous similar messages [ 5198.882174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.895372] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5198.903023] Lustre: Skipped 7 previous similar messages [ 5200.011907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.889304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.825346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.857450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.725222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.859148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.867342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.927942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.914920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.904959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.767520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.631988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.487897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.600750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.595742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.594896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.683093] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5215.695032] Lustre: Skipped 16 previous similar messages [ 5215.746801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.762159] Lustre: Skipped 16 previous similar messages [ 5215.767402] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5215.772308] Lustre: Skipped 16 previous similar messages [ 5215.775740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.788122] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5215.807143] Lustre: Skipped 16 previous similar messages [ 5216.677276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.676199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.492984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.431855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.296171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.302347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.230850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.208313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.170759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.242702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.257439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.166595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.075643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.913712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.976963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.878430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.729611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.572035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.720984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.542732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.672074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.693543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.804790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.630367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.589800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.419737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.364353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.178549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.956712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.651906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.540298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.345343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.339800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.244781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.374346] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5248.379629] Lustre: Skipped 34 previous similar messages [ 5248.398884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5248.408077] Lustre: Skipped 34 previous similar messages [ 5248.410259] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5248.413256] Lustre: Skipped 34 previous similar messages [ 5248.415221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.421650] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5248.430373] Lustre: Skipped 34 previous similar messages [ 5249.354597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.382851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.550024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.352636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.599606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.786710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.592261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.488194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.125671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.826396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.362294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.224520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.940072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.848939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.765642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.786925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.831590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.869658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.758185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.760594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.776226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.557672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.459270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.416456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.221891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.212477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.813572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.550300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.371699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.359047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.290618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.317758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.160695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.051382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.789840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.658058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.352228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.309763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.417414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.549795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.445190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.239314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.066926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.062896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.905823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.979306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.778226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.961081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.216581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.336838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.305669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.623978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.860440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.105993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.986176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.804479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.824278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.181052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.624847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.863518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.048880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.223051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.187852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.488803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.373845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.328088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.428979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5313.435288] Lustre: Skipped 66 previous similar messages [ 5313.446057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5313.454311] Lustre: Skipped 66 previous similar messages [ 5313.460760] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5313.463829] Lustre: Skipped 66 previous similar messages [ 5313.467729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.481476] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5313.484013] Lustre: Skipped 66 previous similar messages [ 5314.398494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.512656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.297559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.361537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.357926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.986965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.467245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.975894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.774064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.577952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.735824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.780614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.733208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.646252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.494340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.407123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.142179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.041992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.988159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.182291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.101570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.079348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.903909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.853789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.891484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.734815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.424255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.196919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.018867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.033718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.754981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.641077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.340223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.213301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.922412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.888915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.680171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.527874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.507777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.446423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.147775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.042309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.998067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.924554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.856944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.844691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.969766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.842920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.631437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.470218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.354903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.331362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.224254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.978734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.970559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.813177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.896544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.624645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.527431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.249167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.182860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.948951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.926406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.736701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.780545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.617117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.548134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.529850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.420406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.472954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.406713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.253989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.214509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.138477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.043314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.928980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.979484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.840974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.957723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.675394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.675092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.487917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.439645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.410752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.474427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.597513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.582309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.674640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.540306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.674347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.495693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.505506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.331168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.546662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.399862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.539373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.731732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.587422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.727963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.571475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.627272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.426077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.426335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.450503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.481522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.576663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.565071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.558521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.426650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.257610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.395300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.498675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.510393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.498157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.415572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.383868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.251965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.297892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.198263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.304274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.354220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.282075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.162065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.093539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.981462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.880500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.774880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.683217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.568483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.362722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.175729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.061306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.935684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.028201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.939948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.032336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.895306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.918661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.713582] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5441.717773] Lustre: Skipped 138 previous similar messages [ 5441.733795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.741269] Lustre: Skipped 138 previous similar messages [ 5441.751926] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5441.757435] Lustre: Skipped 138 previous similar messages [ 5441.761628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.770758] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5441.773380] Lustre: Skipped 138 previous similar messages [ 5442.756267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.713446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.601302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.594861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.454324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.414708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.408098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.539876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.509514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.515178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.523132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.556599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.732457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.669695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.557739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.523700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.697216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.928901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.067549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.804462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.751384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.605051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.493389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.325337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.188189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.957369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.919026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.793089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.714097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.640611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.556524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.397408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.495181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.347926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.546829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.367310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.142879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.099031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.999331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.788384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.556759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.457025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.648501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.655879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.747321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.575827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.501156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.337739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.223112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.108528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.020134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.054858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.087550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.198593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.337072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.237114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.261936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.078367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.158920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.012989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.989751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.938500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.177993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.250838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.305640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.403861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.371899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.315825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.266456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.281715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.326757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.237573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.037182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.923574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.689779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.612687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.452608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.366153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.173774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.163837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.055385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.880451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.616753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.480975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.318969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.092680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.032482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.900790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.730271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.685030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.659527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.281340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.164246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.179892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.005918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.985827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.854971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.885367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.971780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.075460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.807918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.904119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.858260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.824474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.899735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.346463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.572496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.442962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.805216] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5545.819517] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5545.835109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.098458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.447360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.029363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.086183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.087982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.136694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.037543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.205803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.047115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.106098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.074798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.113778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.946231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.770683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.453266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.138807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.697375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.393517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.351101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.997410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.632846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.499183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.349383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.316695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.250720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.472964] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5570.483236] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5570.488140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.455818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.328730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.589278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.534756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.324209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.229202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.487027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.523693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.485575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.396489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.219084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.127894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.092710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.008832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.182908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.904939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.838824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.701749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.624098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.671472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.642917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.462142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.378595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.239781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.265296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.350250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.114627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.068123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.157298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.143995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.990558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.916667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.959877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.929145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.930176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.848978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.902511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.666149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.469467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.193725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.126429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.109433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.100608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.979788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.073558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.937474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.087987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.043730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.840923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.608846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.607510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.634439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.626585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.596230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.739717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.740830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.535605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.398382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.403044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.553522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.471092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.386852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.314855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.205302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.194828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.972567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.060060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.855372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.868271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.831794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.845095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.538570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.448459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.234699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.065460] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5641.077644] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5641.081823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.087742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.055221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.870683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.759598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.617884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.652717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.511963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.536029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.595493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.515096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.345679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.231729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.249067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.402443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.352645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.412451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.549634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.356991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.227364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.154161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.150732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.131726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.949673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.741060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.730847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.522438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.404594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.208949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.166993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.149545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.980793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.824084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.819901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.822669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.932319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.853729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.707347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.665285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.657056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.623300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.430467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.256334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.198991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.243168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.316927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.277068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.327921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.329605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.432565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.474621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.444350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.452826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.577984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.746544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.867211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.904376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.041852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.095102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.916974] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5697.922883] Lustre: Skipped 268 previous similar messages [ 5697.941950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.950520] Lustre: Skipped 268 previous similar messages [ 5697.954571] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5697.958744] Lustre: Skipped 268 previous similar messages [ 5697.963284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.981415] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5697.987510] Lustre: Skipped 268 previous similar messages [ 5699.031401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.062810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.171593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.154757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.145190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.100032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.219979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.420277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.618992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.672498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.510812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.595079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.558099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.488466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.432932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.243023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.173660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.208099] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5716.214815] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5716.226636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.187847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.095126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.006613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.047026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.036301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.175046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.097674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.147348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.085389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.254612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.216748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.266753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.215874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.418633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.265324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.479359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.486994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.422057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.349774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.428851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.267421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.319152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.029768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.955631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.807366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.824176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.685488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.702157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.550830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.444859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.417099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.230540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.922071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.688671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.593333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.475825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.438448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.951727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.907417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.007650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.901270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.791356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.734301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.771054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.698194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.886246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.000534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.939707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.954872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.848257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.776324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.919994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.783507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.900961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.015324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.681721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.766350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.975037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.201249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.502700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.386305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.573089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.818623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.176310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.203163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.486577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.693237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.973681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.211516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.636120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.630751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.407371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.235205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.223690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.933350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.894280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.612250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.825158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.855659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.597877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.294852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.027084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.044923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.227579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.239281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.195571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.960831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.751401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.770958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.156612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.190312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.246502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.994448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.563658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.487969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.235870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.251718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.211464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.319580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.017260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.304720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.124538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.814991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.227995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.064410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.981864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.992177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.056251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.021400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.997597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.884243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.997373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.926066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.877420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.603189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.666553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.668180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.752786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.672192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.559683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.427454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.255934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.209987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.082843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.037774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.888819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.908074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.952882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.980982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.046694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.928523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.831372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.630368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.553580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.646126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.700529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.907756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.822817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.738978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.659029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.572275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.578665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.483482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.453368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.350358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.303703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.270242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.273326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.199171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.126832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.904552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.770470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.914172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.923313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.935722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.788151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.879736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.009459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.022459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.079664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.085029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.102045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.241706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.110468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.166480] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5879.174413] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5879.203725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.084942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.984600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.808426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.674281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.549199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.495705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.449937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.387218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.271708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.184977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.046724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.058856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.790348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.850182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.963912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.767327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.898629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.837691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.702587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.817940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.109957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.122241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.957630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.874202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.856825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.689594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.755922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.680604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.590466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.354261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.332223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.282509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.379696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.443309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.401662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.523837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.612812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.658700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.699487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.743635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.629335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.535637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.252556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.231356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.076915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.192811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.267879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.357825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.323313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.304723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.252151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.275399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.149112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.130743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.959753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.757313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.449966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.335318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.043469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.115497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.962744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.823524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.932913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.914859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.933576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.879969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.705529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.741240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.732931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.701866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.601697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.620555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.559971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.502636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.494397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.240044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.082549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.190736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.973897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.858945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.640902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.595167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.554722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.497850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.334219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.292864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.138811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.048867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.965476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.821490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.796859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.924991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.870772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.769039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.570479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.507157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.396545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.269343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.317261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.294468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.529059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.351965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.322566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.341282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.349708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.307810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.274561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.213173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.207614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.009372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.096353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.245505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.130797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.142624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.027683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.969745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.710887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.733646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.521416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.456844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.471326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.501329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.365417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.450696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.342173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.162569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.199989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.014377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.933752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.807853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.727624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.719914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.691735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.623546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.314445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.218907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.071513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.036460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.942476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.902720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.645971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.476512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.321687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.414340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.285790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.211517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.093231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.289303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.166595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.269835] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6020.289996] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6020.298191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.070313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.007610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.956298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.920848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.787337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.635428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.464419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.442672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.400963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.543169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.466249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.403818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.258673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.225312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.087528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.903871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.787123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.637676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.566286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.539311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.286097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.200239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.152185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.029835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.772606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.880793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.675235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.537722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.449668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.463621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.474696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.357971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.489852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.361958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.304874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.316766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.225267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.121885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.975623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.800250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.860697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.986774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.974941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.943938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.967069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.866586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.869316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.802627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.414870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.268510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.109070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.943630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.906750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.834301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.745430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.885416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.026548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.008929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.225234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.287741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.314876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.495420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.780120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.213323] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6081.220208] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6081.234255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.150573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.247349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.110644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.329817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.188341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.236904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.235146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.235835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.334950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.204048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.594455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.556156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.706885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.659661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.680896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.650592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.634905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.599338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.592742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.641934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.604217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.527097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.591197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.552369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.511902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.375236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.420501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.412496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.385996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.303807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.276364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.420220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.504755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.402162] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6115.418813] LustreError: 172121:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6115.433017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.622383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.911085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.114720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.176654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.425549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.263071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.263100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.115216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.005108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.788925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.865827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.615311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.521491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.422448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.300976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.187887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.398804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.464221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.408811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.267991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.884035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.684062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.425459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.382460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.988446] LustreError: 173923:0:(ofd_dev.c:1784:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6457.003046] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6457.014268] LustreError: 172121:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 6885.360631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.381600] Lustre: Skipped 455 previous similar messages [ 6885.386742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6885.394402] Lustre: Skipped 1 previous similar message [ 6890.471120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6890.481884] Lustre: Skipped 1 previous similar message [ 6891.024703] Lustre: server umount lustre-OST0000 complete [ 6895.586476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6895.590599] Lustre: Skipped 1 previous similar message [ 6900.708536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6900.714989] Lustre: Skipped 1 previous similar message [ 6901.244178] Lustre: server umount lustre-MDT0000 complete [ 6905.112485] LustreError: 172081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872508 with bad export cookie 13276489206328665747 [ 6905.116400] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6905.141559] LustreError: 172081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6905.968319] Lustre: server umount lustre-MDT0001 complete [ 6917.576496] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6920.259288] Key type lgssc unregistered [ 6920.526508] LNet: 227337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6920.534212] LNetError: 227337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6920.551825] LNet: Removed LNI 192.168.201.121@tcp [ 6921.399373] Key type .llcrypt unregistered [ 6921.401196] Key type ._llcrypt unregistered [ 6946.621977] Key type ._llcrypt registered [ 6946.624332] Key type .llcrypt registered [ 6946.744833] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6948.437975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.459087] alg: No test for adler32 (adler32-zlib) [ 6949.499353] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6949.846502] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6951.591352] Key type lgssc registered [ 6953.108754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6961.612301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6961.666273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.287697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.210625] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6973.409476] LustreError: 228752:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.425070] LustreError: 228752:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6977.232946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.639357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6982.596866] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6989.862930] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6994.572879] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7001.195854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.598030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.240994] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7008.915342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49664 to 0x280000401:49695) [ 7016.495997] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.920232] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7023.551781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.561254] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7023.565268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7023.571819] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7023.574213] LustreError: 228773:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7028.192813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.206393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.235205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.226949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.232542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.259888] Lustre: Skipped 1 previous similar message [ 7031.431350] Lustre: server umount lustre-OST0000 complete [ 7043.653146] LustreError: 228773:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7067.483545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7067.867693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.937339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7069.985335] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7075.445963] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7079.230276] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7079.254630] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7083.657543] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.284901] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7090.136860] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d0545ea3000.ost_server_uuid 50 [ 7091.931369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d0545ea3000.ost_server_uuid in IDLE state after 0 sec [ 7097.312802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7097.318049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.326778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.330919] Lustre: Skipped 1 previous similar message [ 7099.872837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7099.877511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7099.897212] Lustre: Skipped 1 previous similar message [ 7101.831292] Lustre: server umount lustre-OST0000 complete [ 7110.116723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.122683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7110.127494] Lustre: Skipped 2 previous similar messages [ 7110.141590] Lustre: Skipped 1 previous similar message [ 7112.294401] Lustre: server umount lustre-MDT0000 complete [ 7115.235389] LustreError: 228749:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.263749] LustreError: 228749:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7117.150328] LustreError: 228732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872720 with bad export cookie 3585069293810069131 [ 7117.160276] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.171453] LustreError: 228732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7117.748391] Lustre: server umount lustre-MDT0001 complete [ 7132.301966] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7136.237579] Key type lgssc unregistered [ 7136.990487] LNet: 233968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7137.012577] LNetError: 233968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7137.030938] LNet: Removed LNI 192.168.201.121@tcp [ 7137.968546] Key type .llcrypt unregistered [ 7137.970217] Key type ._llcrypt unregistered [ 7173.285807] Key type ._llcrypt registered [ 7173.289645] Key type .llcrypt registered [ 7173.400503] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7174.647607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7174.787207] alg: No test for adler32 (adler32-zlib) [ 7175.909209] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 7176.196994] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7177.895474] Key type lgssc registered [ 7179.521406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7189.329980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7189.348427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.720250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.983523] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7200.243638] LustreError: 235702:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.273606] LustreError: 235702:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7205.350725] LustreError: 235701:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.489447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.879096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.523710] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7215.609340] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7222.060352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.353147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.216454] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7236.754945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49697 to 0x280000401:49727) [ 7238.033096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.313165] Lustre: lustre-OST0001: new disk, initializing [ 7238.316934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7238.325450] Lustre: 238009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7238.456764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7238.727487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7238.751913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7238.912130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7245.656429] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7257.189277] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7266.366628] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7280.109178] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.112080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.124879] Lustre: Skipped 3 previous similar messages [ 7280.141909] Lustre: Skipped 3 previous similar messages [ 7285.226229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.232323] Lustre: Skipped 3 previous similar messages [ 7285.825703] Lustre: server umount lustre-MDT0000 complete [ 7289.670285] LustreError: 235683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872892 with bad export cookie 17094759330084845134 [ 7289.677983] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.684439] LustreError: 235683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7290.188074] Lustre: server umount lustre-MDT0001 complete [ 7304.883714] Lustre: server umount lustre-OST0000 complete [ 7306.723904] Lustre: 234963:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872893/real 1787872893] req@ffff97179ab3b840 x1874720487210496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787872909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7306.793750] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.727022] Lustre: server umount lustre-OST0001 complete [ 7332.429873] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7342.889237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.407330] LustreError: 241116:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.525409] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7348.268481] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7348.714597] LustreError: 241117:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.834054] LustreError: 241116:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.675173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.137065] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7363.206627] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7366.331561] Lustre: 242258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7373.254201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7380.404843] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7382.762510] LustreError: 242611:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.798560] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7389.184618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.197878] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49697 to 0x280000401:49759) [ 7389.559616] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7389.563911] Lustre: Skipped 1 previous similar message [ 7396.093429] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7403.862741] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7407.882121] Lustre: 244130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7419.365427] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7419.379138] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.410425] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7424.997455] Lustre: server umount mylustre-MDT0000 complete [ 7425.520155] LustreError: 242633:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.557338] LustreError: 242633:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 7428.634197] LustreError: 244133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873031 with bad export cookie 17094759330084847997 [ 7428.642580] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.652683] LustreError: 244133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7429.147043] Lustre: server umount mylustre-MDT0001 complete [ 7442.940092] Lustre: server umount mylustre-OST0000 complete [ 7456.938659] Lustre: server umount mylustre-OST0001 complete [ 7478.230949] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7487.068873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.390320] LustreError: 246442:0:(ldlm_lib.c:1228:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.455187] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7491.795306] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7501.061819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.202364] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7509.142939] Lustre: 247583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7516.267776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.074311] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7526.834556] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7532.321470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.629598] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49697 to 0x280000401:49791) [ 7539.445917] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7546.580983] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7550.140710] Lustre: 249456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7557.601788] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7557.607473] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.613069] Lustre: Skipped 3 previous similar messages [ 7557.617241] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7557.619604] Lustre: Skipped 4 previous similar messages [ 7563.071321] Lustre: server umount tfs-MDT0000 complete [ 7564.261183] LustreError: 246442:0:(ldlm_lib.c:1228:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.275778] LustreError: 246442:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 7566.694443] LustreError: 246418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873169 with bad export cookie 17094759330084849950 [ 7566.703030] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.713636] LustreError: 246418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7567.114754] Lustre: server umount tfs-MDT0001 complete [ 7580.290272] Lustre: server umount tfs-OST0000 complete [ 7594.152825] Lustre: server umount tfs-OST0001 complete [ 7617.160593] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7627.503686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.031778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.035686] Lustre: Skipped 3 previous similar messages [ 7633.199776] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7642.385248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.561126] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7650.736659] Lustre: 252849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7658.644499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7665.854748] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7667.328389] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7672.447215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49697 to 0x280000401:49823) [ 7674.983887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.926991] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7689.561089] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7693.611417] Lustre: 254721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7700.402543] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 19:28:21 (1787873301) [ 7716.322827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.326841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.341331] Lustre: Skipped 5 previous similar messages [ 7716.345841] Lustre: Skipped 5 previous similar messages [ 7721.981571] Lustre: server umount lustre-MDT0000 complete [ 7725.338487] LustreError: 251683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873328 with bad export cookie 17094759330084851637 [ 7725.349446] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7725.358795] LustreError: 251683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7726.561963] LustreError: 251704:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.562766] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.564912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7726.564922] Lustre: Skipped 6 previous similar messages [ 7726.590584] LustreError: 251704:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 7726.605761] Lustre: Skipped 2 previous similar messages [ 7731.887097] Lustre: server umount lustre-MDT0001 complete [ 7741.290400] Lustre: server umount lustre-OST0000 complete [ 7759.328612] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7759.524764] Lustre: server umount lustre-OST0001 complete [ 7766.124663] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 7777.606841] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7816.039560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.307612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7816.332494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7816.413463] Lustre: lustre-MDT0000: new disk, initializing [ 7816.467730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7816.476459] Lustre: Skipped 3 previous similar messages [ 7816.496103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7821.206777] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7831.960803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.146341] Lustre: 257908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7832.212090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7832.216647] Lustre: Skipped 1 previous similar message [ 7832.297507] Lustre: lustre-MDT0001: new disk, initializing [ 7832.497058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7832.513817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7837.697550] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7849.565577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7849.798624] Lustre: lustre-OST0000: new disk, initializing [ 7849.801883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7849.809119] Lustre: 259489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7851.555659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7851.570897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7851.720745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7857.510137] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7868.648479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.820855] Lustre: lustre-OST0001: new disk, initializing [ 7868.823114] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7868.826570] Lustre: 260362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7870.641571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7870.662631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7870.860969] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7874.985141] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7886.908699] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 19:31:28 (1787873488) [ 7899.250725] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 19:31:40 (1787873500) [ 7917.040158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7917.044279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7917.046955] Lustre: Skipped 1 previous similar message [ 7917.052017] Lustre: Skipped 3 previous similar messages [ 7928.289073] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7928.390609] Lustre: server umount lustre-OST0000 complete [ 7938.823678] Lustre: server umount lustre-MDT0000 complete [ 7942.666644] LustreError: 257898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873545 with bad export cookie 17094759330084853191 [ 7942.669358] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7942.679423] LustreError: 257898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7949.400855] Lustre: server umount lustre-MDT0001 complete [ 7961.205380] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8039.595249] Lustre: server umount lustre-OST0001 complete [ 8048.514162] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 8066.253781] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8106.152083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.430562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8106.484381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8106.575576] Lustre: lustre-MDT0000: new disk, initializing [ 8106.680416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.693259] Lustre: Skipped 3 previous similar messages [ 8106.708846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8111.393087] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8122.145298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.313666] Lustre: 265211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8122.577729] Lustre: lustre-MDT0001: new disk, initializing [ 8122.831858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8122.866923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8127.907510] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8137.848477] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8143.429549] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.038169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8151.492748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8151.500816] Lustre: Skipped 2 previous similar messages [ 8151.513664] Lustre: 267179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8153.468153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8153.626168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8158.965634] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8171.203672] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8179.180640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.185513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.192527] Lustre: Skipped 5 previous similar messages [ 8179.196796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8179.199195] Lustre: Skipped 13 previous similar messages [ 8179.214269] LustreError: Skipped 1 previous similar message [ 8185.271842] Lustre: server umount lustre-OST0000 complete [ 8194.530503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8196.627782] Lustre: server umount lustre-MDT0000 complete [ 8199.667960] LustreError: 265222:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.699940] LustreError: 265222:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 8200.920070] LustreError: 265203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873803 with bad export cookie 17094759330084854668 [ 8200.924260] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8200.940151] LustreError: 265203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8201.249798] Lustre: server umount lustre-MDT0001 complete [ 8213.272935] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8217.624411] Key type lgssc unregistered [ 8217.947091] LNet: 268945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8217.968930] LNetError: 268945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8218.018181] LNet: Removed LNI 192.168.201.121@tcp [ 8219.255698] Key type .llcrypt unregistered [ 8219.257263] Key type ._llcrypt unregistered [ 8252.234885] Key type ._llcrypt registered [ 8252.237516] Key type .llcrypt registered [ 8252.333819] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8253.880893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8253.917403] alg: No test for adler32 (adler32-zlib) [ 8255.036598] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8255.331748] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8257.087386] Key type lgssc registered [ 8258.451624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8267.501288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8267.526976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.235068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.689667] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8279.525775] LustreError: 270406:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.563062] LustreError: 270406:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8284.656821] LustreError: 270407:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8286.587580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8287.074755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8293.932746] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8301.698269] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8307.906156] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8314.028872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.449325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.517164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8321.248093] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8330.791530] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8331.819558] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8336.362870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8336.374773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8336.395141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8337.395870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.399742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8337.407638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8337.432482] Lustre: Skipped 1 previous similar message [ 8340.499589] Lustre: server umount lustre-OST0000 complete [ 8347.617094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.622966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8347.633960] Lustre: Skipped 1 previous similar message [ 8347.657565] Lustre: Skipped 1 previous similar message [ 8350.688293] Lustre: server umount lustre-MDT0000 complete [ 8352.737411] LustreError: 272087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8352.763597] LustreError: 272087:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8354.703505] LustreError: 270387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873957 with bad export cookie 14847243067507521951 [ 8354.704636] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.708972] LustreError: 270387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8354.989215] Lustre: server umount lustre-MDT0001 complete [ 8368.706447] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8372.353644] Key type lgssc unregistered [ 8372.762153] LNet: 273994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8372.776625] LNetError: 273994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8372.794737] LNet: Removed LNI 192.168.201.121@tcp [ 8373.922491] Key type .llcrypt unregistered [ 8373.923947] Key type ._llcrypt unregistered [ 8392.755216] Key type ._llcrypt registered [ 8392.757701] Key type .llcrypt registered [ 8392.899349] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8394.401168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8394.622591] alg: No test for adler32 (adler32-zlib) [ 8395.724961] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8395.921987] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8397.615986] Key type lgssc registered [ 8398.843974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8408.469103] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8408.504692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.224126] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8429.343317] LustreError: 275190:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 8456.031537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.542207] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8469.156271] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8471.090385] Lustre: server umount lustre-OST0000 complete [ 8490.457119] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8493.466635] Key type lgssc unregistered [ 8493.917644] LNet: 276826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8493.943907] LNetError: 276826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8495.013755] LNet: Removed LNI 192.168.201.121@tcp [ 8496.178438] Key type .llcrypt unregistered [ 8496.184063] Key type ._llcrypt unregistered [ 8540.054969] Key type ._llcrypt registered [ 8540.057969] Key type .llcrypt registered [ 8540.139394] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8541.350861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8541.387608] alg: No test for adler32 (adler32-zlib) [ 8542.570656] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 8542.958105] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8544.687226] Key type lgssc registered [ 8545.991718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8560.209305] loop0: detected capacity change from 0 to 204800000 [ 8560.338550] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8560.363311] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8560.670970] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8561.849832] loop0: detected capacity change from 0 to 204800000 [ 8562.106526] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8562.113673] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8562.434637] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8563.493745] loop0: detected capacity change from 0 to 204800000 [ 8563.733529] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8563.740339] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8563.933113] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8565.214324] loop0: detected capacity change from 0 to 204800000 [ 8565.397869] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8565.408616] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8565.578812] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8566.717994] loop0: detected capacity change from 0 to 204800000 [ 8566.756640] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8571.045956] loop0: detected capacity change from 0 to 204800000 [ 8571.069465] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8575.384067] loop0: detected capacity change from 0 to 204800000 [ 8575.417635] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8580.338269] loop0: detected capacity change from 0 to 204800000 [ 8580.366611] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8584.724100] loop0: detected capacity change from 0 to 204800000 [ 8584.792651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8584.833940] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.934716] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8584.943824] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8584.954488] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8584.963589] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8584.972983] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8584.984810] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8584.996393] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8585.008861] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8585.034541] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8585.042374] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8585.053515] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8585.067212] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8585.085731] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8585.116279] Lustre: 280240:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8590.660568] Lustre: server umount MGS complete [ 8591.448361] loop0: detected capacity change from 0 to 204800000 [ 8591.577174] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.469626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.567575] Lustre: lustre-MDT0000: Aborting client recovery [ 8593.588242] loop1: detected capacity change from 0 to 204800000 [ 8593.703950] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.861046] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8593.868698] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8593.882513] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8593.900108] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8593.923313] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8593.941424] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8593.955411] Lustre: 280672:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8593.976558] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8594.220107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8594.318391] Lustre: lustre-MDT0001: Aborting client recovery [ 8595.273551] loop2: detected capacity change from 0 to 204800000 [ 8595.470311] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8595.579492] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8595.593820] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8595.602423] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8595.615669] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8595.628028] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8595.639479] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8595.649864] Lustre: 280750:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8595.836379] Lustre: Found index 0 for lustre-OST0000, updating log [ 8595.893923] Lustre: 280769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8596.016178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8596.051363] Lustre: lustre-OST0000: Aborting recovery [ 8597.128822] loop3: detected capacity change from 0 to 204800000 [ 8597.358807] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.416173] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8597.424298] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8597.432348] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8597.445975] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8597.461605] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8597.470335] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8597.480484] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8597.497703] Lustre: Found index 1 for lustre-OST0001, updating log [ 8597.524890] Lustre: 280840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8597.597946] Lustre: lustre-OST0001: Aborting recovery [ 8602.704870] LustreError: 280616:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8603.628639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8603.654444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8603.665339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8603.677865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8604.652100] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8604.658629] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8606.811054] LustreError: 280583:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 748us: rc = -2 [ 8607.345458] LustreError: 280584:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 21823us: rc = -2 [ 8607.410926] LustreError: 280584:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 8625.126901] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8625.133714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8625.160406] Lustre: Skipped 3 previous similar messages [ 8625.176248] Lustre: Skipped 2 previous similar messages [ 8625.204729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8626.743886] Lustre: server umount lustre-MDT0000 complete [ 8627.779231] LustreError: 280565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874230 with bad export cookie 1871911735883333319 [ 8627.793054] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8630.242991] LustreError: 280580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8630.243336] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.254454] LustreError: 280580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8630.310149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8630.311928] Lustre: Skipped 2 previous similar messages [ 8632.288527] LustreError: 280580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.288802] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8632.298954] LustreError: 280580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8632.312034] Lustre: Skipped 1 previous similar message [ 8634.545449] Lustre: server umount lustre-MDT0001 complete [ 8641.878464] Lustre: server umount lustre-OST0000 complete [ 8648.941436] Lustre: server umount lustre-OST0001 complete [ 8657.600688] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 19:44:18 (1787874258) [ 8688.429806] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 8699.363096] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8735.762766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.014302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8736.062384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.149147] Lustre: lustre-MDT0000: new disk, initializing [ 8736.208372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.216034] Lustre: Skipped 1 previous similar message [ 8736.230893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8741.617358] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8752.441572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8752.618263] Lustre: 284904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8752.682665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8752.687339] Lustre: Skipped 1 previous similar message [ 8752.818956] Lustre: lustre-MDT0001: new disk, initializing [ 8752.977104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8753.049480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8753.077052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.161661] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8770.125773] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8776.413142] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8784.128400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.462214] Lustre: lustre-OST0000: new disk, initializing [ 8784.465295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8784.471585] Lustre: 286876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8784.575032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8790.186607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8790.208514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8790.382683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8792.089439] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8802.636803] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8808.513144] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8819.449873] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8825.730285] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8877.540225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.541467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8877.552859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8877.575626] Lustre: Skipped 2 previous similar messages [ 8882.664068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8882.677626] Lustre: Skipped 1 previous similar message [ 8887.263168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8887.521962] Lustre: server umount lustre-OST0000 complete [ 8892.898340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8892.917161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8892.928379] Lustre: Skipped 2 previous similar messages [ 8906.207127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8906.661682] Lustre: server umount lustre-MDT0000 complete [ 8907.237323] LustreError: 284914:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.261851] LustreError: 284914:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8910.739527] LustreError: 284899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874513 with bad export cookie 1871911735883336028 [ 8910.748882] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8910.759175] LustreError: 284899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8911.064561] Lustre: server umount lustre-MDT0001 complete [ 8920.965745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.622820] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8930.820353] Lustre: server umount MGS complete [ 8943.730883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.273149] LustreError: 290695:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.300790] LustreError: 290708:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8944.363380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.551851] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8949.729682] LustreError: 290696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.623726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8963.488378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8973.019784] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8980.755540] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8988.326679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8988.787963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.793734] Lustre: Skipped 1 previous similar message [ 8996.403446] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9004.570855] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9036.258167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9036.275289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.294851] Lustre: Skipped 1 previous similar message [ 9036.304764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9036.313668] Lustre: Skipped 4 previous similar messages [ 9038.309734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9042.408726] Lustre: server umount lustre-OST0000 complete [ 9048.555557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.609489] Lustre: Skipped 2 previous similar messages [ 9061.343130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9061.805408] Lustre: server umount lustre-MDT0000 complete [ 9063.914244] LustreError: 290692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9063.939742] LustreError: 290692:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9066.023652] LustreError: 290675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874668 with bad export cookie 1871911735883337883 [ 9066.028424] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9066.034987] LustreError: 290675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9066.344397] Lustre: server umount lustre-MDT0001 complete [ 9079.413752] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9082.890949] Key type lgssc unregistered [ 9083.287636] LNet: 294646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9083.313365] LNetError: 294646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9083.343765] LNet: Removed LNI 192.168.201.121@tcp [ 9084.574047] Key type .llcrypt unregistered [ 9084.577120] Key type ._llcrypt unregistered [ 9128.298172] Key type ._llcrypt registered [ 9128.299913] Key type .llcrypt registered [ 9128.499128] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 9148.760913] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9150.789461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9150.879890] alg: No test for adler32 (adler32-zlib) [ 9152.035400] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 9152.506996] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9154.320142] Key type lgssc registered [ 9155.825243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9196.945503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9196.962833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.313362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9198.343641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9198.471235] Lustre: lustre-MDT0000: new disk, initializing [ 9198.547759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9198.581840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9203.844522] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9216.059208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.195121] Lustre: 298610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.269272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9216.272528] Lustre: Skipped 1 previous similar message [ 9216.376581] Lustre: lustre-MDT0001: new disk, initializing [ 9216.536045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9216.614933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9216.622704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9222.398329] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9233.161368] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9239.354385] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9247.881085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9248.168745] Lustre: lustre-OST0000: new disk, initializing [ 9248.170803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9248.181287] Lustre: 300581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9248.322696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9253.948668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9253.963669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9254.113061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9254.389540] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9264.505647] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9272.623282] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9286.506424] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9290.716051] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9295.376915] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9341.413108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.416888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9341.440975] Lustre: Skipped 1 previous similar message [ 9341.452689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9341.464341] Lustre: Skipped 1 previous similar message [ 9344.564927] Lustre: server umount lustre-OST0000 complete [ 9349.600668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9349.606805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.620895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.649252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.651840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.682619] Lustre: Skipped 1 previous similar message [ 9355.291896] Lustre: server umount lustre-MDT0000 complete [ 9359.664778] LustreError: 301437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874962 with bad export cookie 2356203363899218582 [ 9359.672560] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.674520] LustreError: 301437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9360.085701] Lustre: server umount lustre-MDT0001 complete [ 9370.747327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.547641] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9380.570706] Lustre: server umount MGS complete [ 9394.247239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.815885] LustreError: 304453:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.830663] LustreError: 304453:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9394.841915] LustreError: 304466:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9394.914507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9400.290433] LustreError: 304454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.522410] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9404.384609] LustreError: 304453:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9409.509915] LustreError: 304454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.259912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.752155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9415.077926] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9423.860185] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9430.656543] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9438.510743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9438.912204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9446.249503] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9455.189455] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9491.425988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9491.437245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.451667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9491.468422] Lustre: Skipped 1 previous similar message [ 9495.033917] Lustre: server umount lustre-OST0000 complete [ 9501.666113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9501.705494] Lustre: Skipped 1 previous similar message [ 9501.718612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9501.730811] Lustre: Skipped 1 previous similar message [ 9506.788816] LustreError: 307182:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.815949] LustreError: 307182:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9508.186425] Lustre: server umount lustre-MDT0000 complete [ 9512.335928] LustreError: 304435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875115 with bad export cookie 2356203363899220500 [ 9512.343188] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9512.353805] LustreError: 304435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9512.818945] Lustre: server umount lustre-MDT0001 complete [ 9524.840805] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9527.471790] Key type lgssc unregistered [ 9527.696921] LNet: 308407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9527.707385] LNetError: 308407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9527.741072] LNet: Removed LNI 192.168.201.121@tcp [ 9528.590160] Key type .llcrypt unregistered [ 9528.596748] Key type ._llcrypt unregistered [ 9549.132772] Key type ._llcrypt registered [ 9549.134660] Key type .llcrypt registered [ 9549.275201] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 9566.581752] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9567.411368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9567.436146] alg: No test for adler32 (adler32-zlib) [ 9568.819326] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 9569.289730] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9571.112111] Key type lgssc registered [ 9572.406663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9600.166938] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9615.359932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9615.421332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.851925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9616.889945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9617.058206] Lustre: lustre-MDT0000: new disk, initializing [ 9617.239756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.272036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9622.291558] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9633.546890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.794603] Lustre: lustre-OST0000: new disk, initializing [ 9633.800152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9633.805525] Lustre: Skipped 1 previous similar message [ 9633.814875] Lustre: 312461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9633.898464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.233067] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9639.577123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9639.590228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9639.606640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10474.501500] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10474.967805] Lustre: server umount lustre-MDT0000 complete [10478.527620] LustreError: 311647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876081 with bad export cookie 8835650829021090754 [10478.537797] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10478.541264] LustreError: 311647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10480.660921] Lustre: server umount lustre-OST0000 complete [10508.926618] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [10521.697443] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [10558.695916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.966904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10558.999234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10559.157800] Lustre: lustre-MDT0000: new disk, initializing [10559.319964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10559.389618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10564.348282] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10574.965655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10575.047752] Lustre: 316805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10575.097396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10575.101532] Lustre: Skipped 1 previous similar message [10575.158607] Lustre: lustre-MDT0001: new disk, initializing [10575.221537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10575.250266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10575.266595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10578.652883] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10584.814481] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10588.292708] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10594.334867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10594.660478] Lustre: lustre-OST0000: new disk, initializing [10594.665743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10594.686950] Lustre: 318770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10594.863430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10600.668398] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10603.638795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10603.652106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10603.694877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10609.819265] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10614.348593] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10614.524433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10619.630353] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10619.847715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10624.482761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10624.492138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.497693] LustreError: Skipped 1 previous similar message [10624.503155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10624.517798] Lustre: Skipped 1 previous similar message [10624.550442] Lustre: Skipped 1 previous similar message [10628.107503] Lustre: server umount lustre-OST0000 complete [10631.650032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10631.661067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.671216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10634.736759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.739802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10634.799842] Lustre: Skipped 1 previous similar message [10638.215082] Lustre: server umount lustre-MDT0000 complete [10639.843928] LustreError: 316812:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.868693] LustreError: 316812:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10641.844429] LustreError: 316797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876244 with bad export cookie 8835650829024031573 [10641.855182] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.312672] Lustre: server umount lustre-MDT0001 complete [10646.099669] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10647.789509] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 20:17:29 (1787876249) [10654.020538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.427243] LustreError: 321084:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10654.493056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10658.596818] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10659.812352] LustreError: 321085:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.945109] LustreError: 321084:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10666.560190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.951491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10671.864262] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10679.098983] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10684.225933] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10690.154762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10690.515609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10695.908367] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10703.118022] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10711.970703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10712.189203] Lustre: lustre-OST0001: new disk, initializing [10712.196157] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10712.204984] Lustre: 323919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10712.369044] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10719.955367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10719.961698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10720.008801] LustreError: 324239:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10720.521741] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10721.056852] LustreError: 324239:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10721.089104] LustreError: 324239:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10722.093866] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10724.141140] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10724.157956] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10728.310060] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10728.333461] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10731.605991] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10731.842944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10736.620218] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10736.626237] LustreError: 324234:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10737.602803] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d0558884000.ost_server_uuid 50 [10739.073562] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d0558884000.ost_server_uuid in FULL state after 0 sec [10745.818732] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d0558884000.ost_server_uuid 50 [10747.346192] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d0558884000.ost_server_uuid in FULL state after 0 sec [10755.051830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10762.623584] Lustre: lustre-OST0001: Not available for connect from 192.168.201.21@tcp (stopping) [10764.771059] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10764.783901] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.730745] Lustre: lustre-OST0001: Not available for connect from 192.168.201.21@tcp (stopping) [10767.737144] Lustre: Skipped 4 previous similar messages [10768.691719] Lustre: server umount lustre-OST0001 complete [10771.431565] LustreError: 322904:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.445086] LustreError: 322904:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10776.553048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.554519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10776.559660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10776.568348] Lustre: Skipped 2 previous similar messages [10776.588734] LustreError: Skipped 2 previous similar messages [10779.864578] Lustre: server umount lustre-OST0000 complete [10784.738564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10790.102924] Lustre: server umount lustre-MDT0000 complete [10791.909856] LustreError: 321812:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10791.930230] LustreError: 321812:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10794.528068] LustreError: 322926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876397 with bad export cookie 8835650829024032455 [10794.530188] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10794.546694] LustreError: 322926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10794.807449] Lustre: server umount lustre-MDT0001 complete [10804.020433] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [10806.353590] Key type lgssc unregistered [10806.564755] LNet: 326608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10806.570105] LNetError: 326608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10806.592629] LNet: Removed LNI 192.168.201.121@tcp [10807.606347] Key type .llcrypt unregistered [10807.608250] Key type ._llcrypt unregistered [10829.411196] Key type ._llcrypt registered [10829.413524] Key type .llcrypt registered [10829.484588] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [10830.142216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10830.258151] alg: No test for adler32 (adler32-zlib) [10831.328950] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [10831.706750] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [10833.535200] Key type lgssc registered [10835.019466] Lustre: Echo OBD driver; http://www.lustre.org/ [10842.085166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10842.104277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.601860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10847.309171] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10853.859312] LustreError: 328034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.868343] LustreError: 328034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10853.944305] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10859.001687] LustreError: 328033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.414654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10860.798476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10867.109547] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10877.528893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10878.214109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10883.816303] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10885.346921] LustreError: 329713:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.441539] LustreError: 329715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.459539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10888.468186] LustreError: 329715:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10892.612574] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10893.797589] LustreError: 329714:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.824295] LustreError: 329714:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10898.387375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.651654] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10904.064509] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10904.086939] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10911.449197] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10919.458237] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10919.600472] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11026.912206] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11026.918439] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.923608] Lustre: Skipped 1 previous similar message [11026.929533] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11026.933425] Lustre: Skipped 1 previous similar message [11032.035910] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11032.042450] Lustre: Skipped 1 previous similar message [11032.752987] Lustre: server umount lustre-OST0001 complete [11037.181497] LustreError: 329715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11037.195651] LustreError: 329715:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11039.206962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11039.217049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11039.225748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11042.292833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11042.339092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11042.353514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11042.362516] Lustre: Skipped 1 previous similar message [11043.819295] Lustre: server umount lustre-OST0000 complete [11047.397798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11047.403722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11047.409609] Lustre: Skipped 1 previous similar message [11047.437029] Lustre: Skipped 1 previous similar message [11053.702996] Lustre: server umount lustre-MDT0000 complete [11057.646076] LustreError: 328033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11057.691930] LustreError: 328033:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11058.656147] LustreError: 328013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876661 with bad export cookie 14020805571042455106 [11058.668698] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11058.685381] LustreError: 328013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11059.242495] Lustre: server umount lustre-MDT0001 complete [11072.496933] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [11076.524897] Key type lgssc unregistered [11076.726702] LNet: 332992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11076.732531] LNetError: 332992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11076.741656] LNet: Removed LNI 192.168.201.121@tcp [11077.446706] Key type .llcrypt unregistered [11077.448220] Key type ._llcrypt unregistered [11128.172877] Key type ._llcrypt registered [11128.174756] Key type .llcrypt registered [11128.359375] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11130.347631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11130.374920] alg: No test for adler32 (adler32-zlib) [11131.453154] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [11131.691303] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [11133.415283] Key type lgssc registered [11134.812428] Lustre: Echo OBD driver; http://www.lustre.org/ [11142.757290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11142.797751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.217124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11149.355471] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11154.410828] LustreError: 335283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.440781] LustreError: 335283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11159.521770] LustreError: 335284:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.326978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11161.051886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11167.097808] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11169.136708] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11177.086853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11177.762633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11185.383907] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11185.963848] LustreError: 336774:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11190.066434] LustreError: 336777:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11190.076213] LustreError: 336777:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11190.078873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11194.602155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11194.854748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11200.028373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11201.675261] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11211.774800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11221.227618] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11235.818974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11235.832586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11235.834538] Lustre: Skipped 2 previous similar messages [11235.850197] Lustre: Skipped 2 previous similar messages [11239.905884] Lustre: server umount lustre-MDT0000 complete [11240.956017] LustreError: 335283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11240.985136] LustreError: 335283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11243.636071] LustreError: 335265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876846 with bad export cookie 11214280418773293264 [11243.639439] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11243.649328] LustreError: 335265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11244.014685] Lustre: server umount lustre-MDT0001 complete [11257.764920] Lustre: server umount lustre-OST0000 complete [11261.408485] Lustre: 334547:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876848/real 1787876848] req@ffff9717a2d3ed00 x1874724634298624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787876864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11261.441800] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.456137] Lustre: Skipped 1 previous similar message [11262.003474] Lustre: server umount lustre-OST0001 complete [11277.149721] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11289.392213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.100589] LustreError: 340552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11290.266161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11294.476233] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11302.673749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11303.085954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11308.225953] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11311.093600] Lustre: 341692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11318.454173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11319.105700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11325.838465] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11327.290090] LustreError: 342048:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.337975] LustreError: 342048:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11334.421975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11335.884469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11335.909974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11342.653328] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11350.706604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11355.123426] Lustre: 343565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11361.783341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.799772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11361.811080] Lustre: Skipped 1 previous similar message [11366.724570] Lustre: server umount lustre-MDT0000 complete [11366.887936] LustreError: 340548:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.931690] LustreError: 340548:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11371.631795] LustreError: 341294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876974 with bad export cookie 11214280418773294965 [11371.653857] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.013939] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.033094] Lustre: Skipped 2 previous similar messages [11372.202545] Lustre: server umount lustre-MDT0001 complete [11378.407146] Lustre: server umount lustre-OST0000 complete [11382.270346] Lustre: server umount lustre-OST0001 complete [11385.933791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11388.246286] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11389.678551] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 20:29:51 (1787876991) [11421.893148] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11431.610388] loop0: detected capacity change from 0 to 1099511627776 [11431.964220] SGI XFS with ACLs, security attributes, quota, no debug enabled [11431.968662] *** VALIDATE xfs *** [11431.994487] XFS (loop0): Mounting V5 Filesystem [11432.040924] XFS (loop0): Ending clean mount [11433.409512] loop1: detected capacity change from 0 to 18691697672192 [11545.131855] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [11553.348079] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11601.783303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11602.440549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11602.486618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11602.670318] Lustre: lustre-MDT0000: new disk, initializing [11602.882677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11602.897614] Lustre: Skipped 1 previous similar message [11602.939290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11607.746588] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11621.642358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11621.793713] Lustre: 348698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11621.857150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11621.865616] Lustre: Skipped 1 previous similar message [11621.952291] Lustre: lustre-MDT0001: new disk, initializing [11622.254583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11622.264848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11626.620487] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11636.922199] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11642.740290] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11649.987381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11650.366808] Lustre: lustre-OST0000: new disk, initializing [11650.378444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11650.387577] Lustre: 350670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11651.749225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11651.774519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11651.907240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11656.798402] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11665.580169] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11671.604461] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11671.873609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11679.743262] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11680.108390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11687.915215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.917157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11687.922845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11687.922855] Lustre: Skipped 3 previous similar messages [11687.948495] Lustre: Skipped 2 previous similar messages [11687.967688] LustreError: Skipped 1 previous similar message [11689.167528] Lustre: server umount lustre-OST0000 complete [11693.025521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.028693] Lustre: Skipped 1 previous similar message [11698.144765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11698.158407] Lustre: Skipped 1 previous similar message [11707.359545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11707.362103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11707.383243] Lustre: Skipped 3 previous similar messages [11707.543422] Lustre: server umount lustre-MDT0000 complete [11711.230623] LustreError: 348692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877314 with bad export cookie 11214280418773296519 [11711.237203] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11711.240508] LustreError: 348692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11711.545454] Lustre: server umount lustre-MDT0001 complete [11718.395837] XFS (loop0): Unmounting Filesystem [11723.076756] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 20:35:24 (1787877324) [11730.197267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11730.555815] LustreError: 353229:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11730.573407] LustreError: 353229:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11730.653504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11730.664605] Lustre: Skipped 2 previous similar messages [11734.783326] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11744.551883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11749.452698] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11757.516115] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11763.711542] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11770.844368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11779.009431] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11787.438734] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11800.036668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.052389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.053878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11800.077407] Lustre: Skipped 3 previous similar messages [11800.104876] Lustre: Skipped 1 previous similar message [11804.359446] Lustre: server umount lustre-OST0000 complete [11814.567360] Lustre: server umount lustre-MDT0000 complete [11818.547180] LustreError: 353209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877421 with bad export cookie 11214280418773297401 [11818.563632] LustreError: 353209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11818.563735] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11818.873639] Lustre: server umount lustre-MDT0001 complete [11828.791504] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [11831.388791] Key type lgssc unregistered [11831.688840] LNet: 356861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11831.700383] LNetError: 356861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.729447] LNet: Removed LNI 192.168.201.121@tcp [11832.748157] Key type .llcrypt unregistered [11832.749811] Key type ._llcrypt unregistered [11864.273778] Key type ._llcrypt registered [11864.276266] Key type .llcrypt registered [11864.438506] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11866.033327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11866.076914] alg: No test for adler32 (adler32-zlib) [11867.196722] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [11867.406989] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [11869.079175] Key type lgssc registered [11870.323967] Lustre: Echo OBD driver; http://www.lustre.org/ [11878.215322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11878.234390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11879.702856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11884.135403] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11889.122440] LustreError: 358431:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.148779] LustreError: 358431:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11893.336572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.761796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11898.810605] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11906.543654] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11912.107092] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11918.158650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11918.593687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11925.191521] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11932.410206] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11944.333374] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.21@tcp (stopping) [11944.418083] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.603639] Lustre: server umount lustre-MDT0001 complete [11949.423263] LustreError: 360924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.545425] LustreError: 358428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.605053] LustreError: 358428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11959.662904] LustreError: 358426:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11959.694981] LustreError: 358426:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11962.586151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11962.858992] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11962.864760] Lustre: Found index 1 for lustre-MDT0001, updating log [11962.943155] Lustre: 358419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11963.268486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11968.506868] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11969.569931] LustreError: 361870:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11969.579630] LustreError: 361870:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11969.597861] LustreError: 358423:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11969.748997] LustreError: 361882:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -17 [11969.762372] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11969.765455] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11969.765455] [11980.189619] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.21@tcp (stopping) [11980.193614] Lustre: Skipped 2 previous similar messages [11983.844633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.851619] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11983.857238] Lustre: Skipped 1 previous similar message [11983.858476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11983.873446] Lustre: Skipped 1 previous similar message [11985.717163] Lustre: server umount lustre-MDT0001 complete [11988.971633] LustreError: 358431:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.993838] LustreError: 358431:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11997.148186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11997.262882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11997.273241] Lustre: Found index 1 for lustre-MDT0001, updating log [11997.332819] Lustre: 358419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11997.740694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12002.816978] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12003.336873] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12003.349474] LustreError: 362953:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12003.359601] LustreError: 362953:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12003.376238] LustreError: 358423:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12003.389929] LustreError: 358423:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12003.558909] LustreError: 362960:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12003.577235] LustreError: 362960:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12003.593729] LustreError: 362960:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -17 [12003.630660] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.121@tcp [12003.630660] [12310.000546] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12310.004547] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12310.066590] Lustre: Skipped 1 previous similar message [12310.091099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12310.098012] Lustre: Skipped 2 previous similar messages [12315.122644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12315.134447] Lustre: Skipped 2 previous similar messages [12323.216056] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.21@tcp (stopping) [12323.238907] Lustre: Skipped 3 previous similar messages [12323.295334] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12323.416396] Lustre: server umount lustre-MDT0001 complete [12325.345919] LustreError: 358428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12325.375334] LustreError: 358428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12339.880268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12340.149441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12340.176035] Lustre: Found index 1 for lustre-MDT0001, updating log [12340.316716] Lustre: 358421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12340.705336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12340.727216] Lustre: Skipped 1 previous similar message [12341.155121] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12346.346719] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12347.910574] LustreError: 371441:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12347.926270] LustreError: 371441:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12347.938742] LustreError: 358423:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12347.949630] LustreError: 358423:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12348.060959] LustreError: 371452:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12348.089849] LustreError: 371452:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -114 [12348.115158] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12348.115158] [12348.550633] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12674.019310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12674.022445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.027586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12674.060710] Lustre: Skipped 3 previous similar messages [12684.255833] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12684.541912] Lustre: server umount lustre-MDT0000 complete [12688.969048] LustreError: 358413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878291 with bad export cookie 4017068113772856733 [12688.969658] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12688.973593] LustreError: 358413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12689.377787] LustreError: 358427:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12689.379315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12689.391988] LustreError: 358427:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [12695.315887] Lustre: server umount lustre-MDT0001 complete [12700.157966] Lustre: server umount lustre-OST0000 complete [12709.662637] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [12719.564446] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [12755.657960] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 20:52:37 (1787878357) [12762.980849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12763.451155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12763.533570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12763.643529] Lustre: lustre-MDT0000: new disk, initializing [12763.739099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12763.776850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12769.994505] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12781.720812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12781.856410] Lustre: 382016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12781.942626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12781.947796] Lustre: Skipped 1 previous similar message [12782.029535] Lustre: lustre-MDT0001: new disk, initializing [12782.200617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12782.236791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12782.250828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12787.426038] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12796.904449] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12803.323559] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12809.113322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12809.361577] Lustre: lustre-OST0000: new disk, initializing [12809.366324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12809.374404] Lustre: 383982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12809.468656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12817.066285] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12818.061940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12818.093305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12818.355234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12829.088769] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12837.866701] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12837.881873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12837.894434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12837.897104] Lustre: Skipped 10 previous similar messages [12842.005989] Lustre: server umount lustre-MDT0000 complete [12844.005870] LustreError: 382021:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12844.017489] LustreError: 382021:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12845.614320] LustreError: 382818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878448 with bad export cookie 4017068113772858686 [12845.620066] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12845.620157] LustreError: 382821:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12845.620414] LustreError: 382821:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003476e8a7 x1874725406293504/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12845.622181] LustreError: 382818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12846.257818] Lustre: server umount lustre-MDT0001 complete [12862.002857] Lustre: server umount lustre-OST0000 complete [12895.093990] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [12906.783622] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [12942.295550] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12634 sec ========= 20:55:42 (1787878542) [12944.426920] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:55:45 (1787878545) === [12949.031826] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:55:49 (1787878549) === [12991.611459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [12994.749299] Key type lgssc unregistered [12995.052643] LNet: 390355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12995.060426] LNetError: 390355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12995.075933] LNet: Removed LNI 192.168.201.121@tcp [12996.017401] Key type .llcrypt unregistered [12996.019371] Key type ._llcrypt unregistered