[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 702930626 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003235] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008310] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011178] LSM: Security Framework initializing [ 0.013047] Yama: becoming mindful. [ 0.014000] SELinux: Initializing. [ 0.014090] *** VALIDATE selinux *** [ 0.022161] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027291] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028192] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030156] *** VALIDATE tmpfs *** [ 0.031554] *** VALIDATE proc *** [ 0.033062] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.035250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038000] Spectre V2 : User space: Vulnerable [ 0.038000] Speculative Store Bypass: Vulnerable [ 0.040742] debug: unmapping init [mem 0xffffffffab259000-0xffffffffab260fff] [ 0.042264] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043792] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051365] rcu: Hierarchical SRCU implementation. [ 0.053829] smp: Bringing up secondary CPUs ... [ 0.054599] x86: Booting SMP configuration: [ 0.055036] .... node #0, CPUs: #1 #2 #3 [ 0.058651] smp: Brought up 1 node, 4 CPUs [ 0.059853] smpboot: Max logical packages: 1 [ 0.060014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.200585] node 0 deferred pages initialised in 136ms [ 0.205247] devtmpfs: initialized [ 0.208329] x86/mm: Memory block size: 128MB [ 0.210755] gcov: version magic: 0x41383552 [ 0.213534] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.217104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218335] pinctrl core: initialized pinctrl subsystem [ 0.219271] [ 0.219588] ************************************************************* [ 0.220024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221032] ** ** [ 0.222019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.223016] ** ** [ 0.224018] ** This means that this kernel is built to expose internal ** [ 0.225020] ** IOMMU data structures, which may compromise security on ** [ 0.226016] ** your system. ** [ 0.227020] ** ** [ 0.228068] ** If you see this message and you are not debugging the ** [ 0.229013] ** kernel, report this immediately to your vendor! ** [ 0.230012] ** ** [ 0.231017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232024] ************************************************************* [ 0.233938] NET: Registered protocol family 16 [ 0.234570] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.235072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.236142] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.237876] cpuidle: using governor menu [ 0.240033] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.242751] PCI: Using configuration type 1 for base access [ 0.244222] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.256092] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.257028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.260093] cryptd: max_cpu_qlen set to 1000 [ 0.262354] ACPI: Added _OSI(Module Device) [ 0.263021] ACPI: Added _OSI(Processor Device) [ 0.264014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269989] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.273665] ACPI: Interpreter enabled [ 0.274094] ACPI: PM: (supports S0 S3 S4 S5) [ 0.275019] ACPI: Using IOAPIC for interrupt routing [ 0.276133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.277562] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.287179] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.288052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292643] acpiphp: Slot [2] registered [ 0.293159] acpiphp: Slot [5] registered [ 0.294155] acpiphp: Slot [6] registered [ 0.295190] acpiphp: Slot [7] registered [ 0.296177] acpiphp: Slot [8] registered [ 0.297168] acpiphp: Slot [9] registered [ 0.298217] acpiphp: Slot [10] registered [ 0.299197] acpiphp: Slot [3] registered [ 0.300133] acpiphp: Slot [4] registered [ 0.301187] acpiphp: Slot [11] registered [ 0.302131] acpiphp: Slot [12] registered [ 0.303173] acpiphp: Slot [13] registered [ 0.304133] acpiphp: Slot [14] registered [ 0.305209] acpiphp: Slot [15] registered [ 0.306124] acpiphp: Slot [16] registered [ 0.307128] acpiphp: Slot [17] registered [ 0.308156] acpiphp: Slot [18] registered [ 0.309126] acpiphp: Slot [19] registered [ 0.310196] acpiphp: Slot [20] registered [ 0.311118] acpiphp: Slot [21] registered [ 0.312129] acpiphp: Slot [22] registered [ 0.313118] acpiphp: Slot [23] registered [ 0.314176] acpiphp: Slot [24] registered [ 0.315118] acpiphp: Slot [25] registered [ 0.316117] acpiphp: Slot [26] registered [ 0.317184] acpiphp: Slot [27] registered [ 0.318196] acpiphp: Slot [28] registered [ 0.319122] acpiphp: Slot [29] registered [ 0.320132] acpiphp: Slot [30] registered [ 0.321207] acpiphp: Slot [31] registered [ 0.322070] PCI host bridge to bus 0000:00 [ 0.323023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329235] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331422] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.333632] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339694] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342359] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.344026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346682] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347933] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.348046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.349754] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.366020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.371020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.376526] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.436024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.457023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470856] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.481019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.491035] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533524] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.540020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.565022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.576879] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.587022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.595025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.618024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.639000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.695941] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.700450] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.705519] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.707274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.712193] iommu: Default domain type: Passthrough [ 0.715766] SCSI subsystem initialized [ 0.717121] ACPI: bus type USB registered [ 0.718189] usbcore: registered new interface driver usbfs [ 0.720099] usbcore: registered new interface driver hub [ 0.722178] usbcore: registered new device driver usb [ 0.724244] pps_core: LinuxPPS API ver. 1 registered [ 0.725011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729073] PTP clock support registered [ 0.731102] EDAC MC: Ver: 3.0.0 [ 0.734869] PCI: Using ACPI for IRQ routing [ 0.735740] NetLabel: Initializing [ 0.737018] NetLabel: domain hash size = 128 [ 0.739014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.741098] NetLabel: unlabeled traffic allowed by default [ 0.744285] vgaarb: loaded [ 0.746483] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.749017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.754640] clocksource: Switched to clocksource kvm-clock [ 0.887923] VFS: Disk quotas dquot_6.6.0 [ 0.889077] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.890548] *** VALIDATE ramfs *** [ 0.891370] *** VALIDATE hugetlbfs *** [ 0.892441] pnp: PnP ACPI init [ 0.894930] pnp: PnP ACPI: found 6 devices [ 0.910902] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.914395] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.916586] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.918729] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.921154] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.923620] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.926466] NET: Registered protocol family 2 [ 0.929337] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.934078] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.936532] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.940642] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.943307] TCP: Hash tables configured (established 65536 bind 65536) [ 0.945619] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.947701] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949727] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951847] NET: Registered protocol family 1 [ 0.953730] RPC: Registered named UNIX socket transport module. [ 0.955283] RPC: Registered udp transport module. [ 0.956946] RPC: Registered tcp transport module. [ 1.083754] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.087987] NET: Registered protocol family 44 [ 1.090102] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.095873] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.099514] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.103221] PCI: CLS 0 bytes, default 64 [ 1.106395] Unpacking initramfs... [ 2.856334] debug: unmapping init [mem 0xffff8a28fcc54000-0xffff8a28fffbffff] [ 2.860493] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.863136] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.866434] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.402445] Initialise system trusted keyrings [ 3.404099] Key type blacklist registered [ 3.405942] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.414954] zbud: loaded [ 3.417994] *** VALIDATE nfs *** [ 3.419156] *** VALIDATE nfs4 *** [ 3.420669] pstore: using deflate compression [ 3.424728] Platform Keyring initialized [ 3.542497] NET: Registered protocol family 38 [ 3.544165] Key type asymmetric registered [ 3.545914] Asymmetric key parser 'x509' registered [ 3.547746] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.550672] io scheduler mq-deadline registered [ 3.552164] io scheduler kyber registered [ 3.553931] io scheduler bfq registered [ 3.556984] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.561817] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.564791] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.567934] ACPI: Power Button [PWRF] [ 3.574645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.585987] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.633605] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.639789] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.652753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.682898] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.713588] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.718450] Non-volatile memory driver v1.3 [ 3.720140] Linux agpgart interface v0.103 [ 3.755704] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.760262] vda: detected capacity change from 0 to 74702848 [ 3.821216] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.825898] vdb: detected capacity change from 0 to 1073741824 [ 3.858070] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.868217] vdc: detected capacity change from 0 to 2621440000 [ 3.889554] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.892068] vdd: detected capacity change from 0 to 2621440000 [ 3.907869] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.910675] vde: detected capacity change from 0 to 4294967296 [ 3.924783] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.926951] vdf: detected capacity change from 0 to 4294967296 [ 3.934274] libphy: Fixed MDIO Bus: probed [ 3.947694] usbcore: registered new interface driver usbserial_generic [ 3.950960] usbserial: USB Serial support registered for generic [ 3.953816] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.961414] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.965764] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.971245] mousedev: PS/2 mouse device common for all mice [ 3.980764] rtc_cmos 00:05: RTC can wake from S4 [ 3.992179] rtc_cmos 00:05: registered as rtc0 [ 3.999672] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.003784] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.022221] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.110957] intel_pstate: CPU model not supported [ 4.117458] hid: raw HID events driver (C) Jiri Kosina [ 4.124561] usbcore: registered new interface driver usbhid [ 4.130335] usbhid: USB HID core driver [ 4.136934] drop_monitor: Initializing network drop monitor service [ 4.138733] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.141082] Initializing XFRM netlink socket [ 4.145346] NET: Registered protocol family 10 [ 4.158659] Segment Routing with IPv6 [ 4.160743] NET: Registered protocol family 17 [ 4.163821] mpls_gso: MPLS GSO support [ 4.172802] RAS: Correctable Errors collector initialized. [ 4.177569] AVX version of gcm_enc/dec engaged. [ 4.180112] AES CTR mode by8 optimization enabled [ 4.270946] sched_clock: Marking stable (4270873735, 0)->(5423050488, -1152176753) [ 4.274195] registered taskstats version 1 [ 4.275894] Loading compiled-in X.509 certificates [ 4.277931] zswap: loaded using pool lzo/zbud [ 4.304871] Key type big_key registered [ 4.318183] Key type encrypted registered [ 4.319564] ima: No TPM chip found, activating TPM-bypass! [ 4.321287] ima: Allocated hash algorithm: sha1 [ 4.322705] ima: No architecture policies found [ 4.324380] evm: Initialising EVM extended attributes: [ 4.326396] evm: security.selinux [ 4.327540] evm: security.ima [ 4.328585] evm: security.capability [ 4.329812] evm: HMAC attrs: 0x1 [ 4.332166] rtc_cmos 00:05: setting system clock to 2026-08-09 19:37:07 UTC (1786304227) [ 4.339225] debug: unmapping init [mem 0xffffffffac203000-0xffffffffac3fffff] [ 4.342381] debug: unmapping init [mem 0xffffffffaaf82000-0xffffffffab258fff] [ 4.350345] Write protecting the kernel read-only data: 28672k [ 4.352916] debug: unmapping init [mem 0xffffffffa9603000-0xffffffffa97fffff] [ 4.355195] debug: unmapping init [mem 0xffffffffa9f14000-0xffffffffa9ffffff] [ 4.392236] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.400895] systemd[1]: Detected virtualization kvm. [ 4.403153] systemd[1]: Detected architecture x86-64. [ 4.405142] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.431167] systemd[1]: No hostname configured. [ 4.432704] systemd[1]: Set hostname to . [ 4.434449] random: systemd: uninitialized urandom read (16 bytes read) [ 4.436421] systemd[1]: Initializing machine ID from random generator. [ 4.541570] random: ln: uninitialized urandom read (6 bytes read) [ 4.707726] random: systemd: uninitialized urandom read (16 bytes read) [ 4.710835] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.728913] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.751836] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.794779] device-mapper: uevent: version 1.0.3 [ 5.798221] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 6.922994] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. [ 6.929982] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 7.538482] scsi host0: ata_piix [ 7.680775] scsi host1: ata_piix [ 7.689316] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.698778] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.838239] random: crng init done [ 11.839596] random: 7 urandom warning(s) missed due to ratelimiting [ 15.018406] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.497480] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.849922] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.812145] SELinux: Disabled at runtime. [ 19.934294] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.980271] systemd[1]: Detected virtualization kvm. [ 19.985280] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.446045] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.459926] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.485674] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.505760] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.518827] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.538729] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.564319] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK [ 22.001898] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 23.460467] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.543298] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.680101] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.667708] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.834219] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 30.850671] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 31.523952] NFS: Registering the id_resolver key type [ 31.529193] Key type id_resolver registered [ 31.530576] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-server login: [ 79.028011] hrtimer: interrupt took 2771524 ns [ 108.323155] libcfs: loading out-of-tree module taints kernel. [ 108.363639] Key type ._llcrypt registered [ 108.365224] Key type .llcrypt registered [ 108.459229] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 129.604392] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 131.523456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.551739] alg: No test for adler32 (adler32-zlib) [ 133.150550] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 133.994619] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 135.863854] Key type lgssc registered [ 137.771180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.280456] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 199.178621] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 213.719298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.768214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.155277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.211669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.351126] Lustre: lustre-MDT0000: new disk, initializing [ 215.487464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.555620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.913298] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.946862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.071412] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 237.094757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 237.106530] Lustre: Skipped 1 previous similar message [ 237.201898] Lustre: lustre-MDT0001: new disk, initializing [ 237.260253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.304103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 237.317572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 242.265493] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.010923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 259.110315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.301775] Lustre: lustre-OST0000: new disk, initializing [ 259.304779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 259.311677] Lustre: 8458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.369788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.743659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.765808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.892908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.692352] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.570267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.716899] Lustre: lustre-OST0001: new disk, initializing [ 280.720240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.725991] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.820652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.437199] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.382928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.406905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 289.521595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 299.558785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.460905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.194440] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.704245] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing yml_node [ 328.970348] Lustre: DEBUG MARKER: Client: 2.17.56.59 [ 331.359543] Lustre: DEBUG MARKER: MDS: 2.17.56.59 [ 334.115309] Lustre: DEBUG MARKER: OSS: 2.17.56.59 [ 335.480057] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 9 15:42:37 EDT 2026 [ 353.338661] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 354.992388] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 371.684187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.686038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.698243] Lustre: Skipped 1 previous similar message [ 371.705955] Lustre: Skipped 3 previous similar messages [ 376.804966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.813279] Lustre: Skipped 3 previous similar messages [ 377.378297] Lustre: server umount lustre-MDT0000 complete [ 381.113473] LustreError: 6512:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786304604 with bad export cookie 1323704616830623890 [ 381.133385] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.920314] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.924280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 381.929753] Lustre: Skipped 3 previous similar messages [ 387.042060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.042807] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.050959] Lustre: Skipped 2 previous similar messages [ 387.076188] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 387.678193] Lustre: server umount lustre-MDT0001 complete [ 392.104842] Lustre: server umount lustre-OST0000 complete [ 396.620968] Lustre: server umount lustre-OST0001 complete [ 404.384617] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 415.119806] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 458.780746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.274839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 459.325159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 459.509117] Lustre: lustre-MDT0000: new disk, initializing [ 459.688350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.769124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 465.229552] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 478.213498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.471590] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 478.618760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 478.633953] Lustre: Skipped 1 previous similar message [ 478.730763] Lustre: lustre-MDT0001: new disk, initializing [ 478.882396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 478.944842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 478.970917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 484.895674] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 496.144927] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.367449] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.323822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.715634] Lustre: lustre-OST0000: new disk, initializing [ 509.718870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 509.726899] Lustre: 17056:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 509.807818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.766780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 511.791971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 511.976162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 516.417982] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 527.196038] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 532.684667] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 532.972490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 538.186897] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 538.416133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 542.693985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.697127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.710538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.733222] Lustre: Skipped 1 previous similar message [ 546.925412] Lustre: server umount lustre-OST0000 complete [ 552.928593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.934639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.944112] Lustre: Skipped 1 previous similar message [ 552.962532] Lustre: Skipped 2 previous similar messages [ 558.051044] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.254642] Lustre: server umount lustre-MDT0000 complete [ 562.156402] LustreError: 15093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.199676] LustreError: 15093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 562.380934] LustreError: 15079:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786304785 with bad export cookie 1323704616830625423 [ 562.402579] LustreError: 15079:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 562.422217] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.910271] Lustre: server umount lustre-MDT0001 complete [ 569.750383] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 15:46:30 (1786304790) [ 579.331256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.214177] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.427981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.701225] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.685495] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 595.938155] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.964345] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 597.246705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.431946] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 612.027822] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 618.230296] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 626.325524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 633.733740] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 643.070082] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 657.951949] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 666.082862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.094549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 666.096557] Lustre: Skipped 1 previous similar message [ 666.111864] Lustre: Skipped 2 previous similar messages [ 671.735805] Lustre: server umount lustre-MDT0000 complete [ 675.539544] LustreError: 19301:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786304898 with bad export cookie 1323704616830626305 [ 675.549441] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.551885] LustreError: 19301:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 675.830263] Lustre: server umount lustre-MDT0001 complete [ 690.805710] Lustre: server umount lustre-OST0000 complete [ 714.055901] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 726.125391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.563163] LustreError: 23727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.657971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.665806] Lustre: Skipped 2 previous similar messages [ 731.142278] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 740.149672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.789987] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 748.031531] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 755.368726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 763.036228] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 772.073229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.294530] Lustre: lustre-OST0001: new disk, initializing [ 772.299697] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 772.305742] Lustre: 26040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 774.196030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 774.214598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 774.356325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 778.832869] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 789.042285] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 793.240942] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 807.396835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 807.414658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.429168] Lustre: Skipped 1 previous similar message [ 807.450408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.463073] Lustre: Skipped 3 previous similar messages [ 812.968712] Lustre: server umount lustre-MDT0000 complete [ 815.590716] LustreError: 23727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.617711] LustreError: 23727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 817.203842] LustreError: 23709:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305040 with bad export cookie 1323704616830627152 [ 817.214580] LustreError: 23709:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 817.216565] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 817.733693] Lustre: server umount lustre-MDT0001 complete [ 833.258384] Lustre: server umount lustre-OST0000 complete [ 836.575179] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786305043/real 1786305043] req@ffff8a2843e1ad80 x1873075877698048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786305059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 836.590104] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.596668] Lustre: Skipped 3 previous similar messages [ 838.114165] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786305045/real 1786305045] req@ffff8a2843e19500 x1873075877698304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786305061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 839.163332] Lustre: server umount lustre-OST0001 complete [ 862.860238] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 874.563063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.336032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.348724] Lustre: Skipped 3 previous similar messages [ 879.819558] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 880.621540] LustreError: 29060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.645407] LustreError: 29060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 888.481502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.642647] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 896.138597] Lustre: 30207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 903.051225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 910.316167] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 919.147249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.589945] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 933.083651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 936.243861] Lustre: 32089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 945.138509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 945.156056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.187226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 945.196459] Lustre: Skipped 4 previous similar messages [ 948.690660] Lustre: server umount lustre-MDT0000 complete [ 952.932908] LustreError: 29041:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305176 with bad export cookie 1323704616830628440 [ 952.944238] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.948892] LustreError: 29041:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 953.290692] Lustre: server umount lustre-MDT0001 complete [ 968.231615] Lustre: server umount lustre-OST0000 complete [ 972.129227] Lustre: 3655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786305179/real 1786305179] req@ffff8a297ef09f80 x1873075877748608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786305195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 972.848041] Lustre: server umount lustre-OST0001 complete [ 982.546698] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 15:53:23 (1786305203) [ 1012.319386] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1023.682286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.384137] LustreError: 34893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.401930] LustreError: 34893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 1030.225698] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1039.960449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.436928] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1047.497546] Lustre: 36044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1055.416035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.530801] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1073.465350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1080.505104] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1089.552630] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1093.314638] Lustre: 37924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1099.750340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.752496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1099.764490] Lustre: Skipped 7 previous similar messages [ 1099.774977] Lustre: Skipped 6 previous similar messages [ 1105.157189] Lustre: server umount lustre-MDT0000 complete [ 1108.703250] LustreError: 34927:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305331 with bad export cookie 1323704616830629553 [ 1108.709476] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.725660] LustreError: 34927:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1109.150202] Lustre: server umount lustre-MDT0001 complete [ 1123.846144] Lustre: server umount lustre-OST0000 complete [ 1126.367152] Lustre: 3655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786305333/real 1786305333] req@ffff8a2843f77b80 x1873075877798784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786305349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1127.845321] Lustre: server umount lustre-OST0001 complete [ 1137.559979] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 15:55:59 (1786305359) [ 1153.185556] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1161.925084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.610190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.615583] Lustre: Skipped 7 previous similar messages [ 1166.527946] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1174.429347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.832372] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1181.209936] Lustre: 41212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1187.050677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.357243] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1201.700386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.787733] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1216.520258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1220.220320] Lustre: 43090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1221.019623] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1221.022521] Lustre: Skipped 1 previous similar message [ 1226.210744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.758694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.771919] Lustre: Skipped 4 previous similar messages [ 1230.499373] Lustre: server umount lustre-MDT0000 complete [ 1234.035965] LustreError: 42402:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305457 with bad export cookie 1323704616830630659 [ 1234.036639] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.044617] LustreError: 42402:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1240.477368] Lustre: server umount lustre-MDT0001 complete [ 1248.645910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.488576] LustreError: 3651:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a297ee4ed80 x1873075877849856/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1265.131111] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1269.491426] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1275.222338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.031274] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1281.043862] Lustre: Skipped 1 previous similar message [ 1281.185752] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1288.776616] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1290.454390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1296.328966] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1297.949516] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1309.026677] Lustre: server umount lustre-MDT0000 complete [ 1309.668609] LustreError: 45710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.694140] LustreError: 45710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 1312.662071] LustreError: 40043:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305535 with bad export cookie 1323704616830631758 [ 1312.663882] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1312.673798] LustreError: 40043:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1312.910021] Lustre: server umount lustre-MDT0001 complete [ 1327.431537] Lustre: server umount lustre-OST0000 complete [ 1329.951094] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786305537/real 1786305537] req@ffff8a294d8aa680 x1873075877887360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786305553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1331.271629] Lustre: server umount lustre-OST0001 complete [ 1339.997847] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 15:59:21 (1786305561) [ 1357.274821] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1368.575815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.586378] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1385.081712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.003090] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1394.076180] Lustre: 48854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1400.882826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.038220] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1418.593517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.651521] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1433.769368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1438.849804] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1449.961122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.966620] Lustre: Skipped 18 previous similar messages [ 1451.975527] LustreError: 50964:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8a29527cfe00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1452.284549] Lustre: server umount lustre-MDT0000 complete [ 1456.421438] LustreError: 47682:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305679 with bad export cookie 1323704616830632584 [ 1456.425077] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.430925] LustreError: 47682:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1456.786771] Lustre: server umount lustre-MDT0001 complete [ 1471.536895] Lustre: server umount lustre-OST0000 complete [ 1485.851476] Lustre: server umount lustre-OST0001 complete [ 1495.413796] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 16:01:57 (1786305717) [ 1498.527366] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1500.412479] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 16:02:01 (1786305721) [ 1510.078137] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 1517.993457] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1539.742733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.956557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1540.013858] Lustre: lustre-MDT0000: new disk, initializing [ 1540.074535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1544.510879] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1554.177405] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1561.302368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1561.569864] Lustre: lustre-OST0000: new disk, initializing [ 1561.572126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1561.575130] Lustre: Skipped 1 previous similar message [ 1561.578656] Lustre: 54943:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1562.769166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1562.778519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1562.842801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1568.279761] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1578.282317] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1581.000828] Lustre: DEBUG MARKER: create test files [ 1613.795498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.807108] Lustre: Skipped 11 previous similar messages [ 1619.717310] Lustre: server umount lustre-OST0000 complete [ 1624.106392] Lustre: server umount lustre-MDT0000 complete [ 1636.451909] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1639.391778] Key type lgssc unregistered [ 1639.680383] LNet: 56579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.691387] LNetError: 56579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.743766] LNet: Removed LNI 192.168.201.125@tcp [ 1642.050225] Key type .llcrypt unregistered [ 1642.058728] Key type ._llcrypt unregistered [ 1677.413268] Key type ._llcrypt registered [ 1677.415082] Key type .llcrypt registered [ 1677.517198] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1678.353115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.396671] alg: No test for adler32 (adler32-zlib) [ 1679.661908] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 1680.052702] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1681.736035] Key type lgssc registered [ 1682.729642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1690.176600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1690.186403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.588269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.842481] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1703.982343] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1711.769743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.217553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.356906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:100 to 0x240000400:129) [ 1716.537411] Lustre: 59314:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1716.546793] Lustre: 59314:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1716.558285] Lustre: 59314:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1716.570915] Lustre: 59314:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1716.575514] Lustre: 59314:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1716.598921] Lustre: 59314:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1719.973041] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1728.922555] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1734.648353] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1740.478605] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1747.427635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.433947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.443805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1749.496592] Lustre: server umount lustre-OST0000 complete [ 1754.088567] Lustre: server umount lustre-MDT0000 complete [ 1766.648635] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1769.849721] Key type lgssc unregistered [ 1770.128303] LNet: 60924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.133070] LNetError: 60924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.178766] LNet: Removed LNI 192.168.201.125@tcp [ 1772.091539] Key type .llcrypt unregistered [ 1772.096984] Key type ._llcrypt unregistered [ 1801.847433] Key type ._llcrypt registered [ 1801.849620] Key type .llcrypt registered [ 1802.000773] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1803.671064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1803.700768] alg: No test for adler32 (adler32-zlib) [ 1804.812955] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 1805.100887] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1806.839159] Key type lgssc registered [ 1808.000204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1816.313841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1816.330793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.628310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.786649] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1828.834165] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1837.258344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.720663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.873601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1842.965748] Lustre: 63740:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1842.972258] Lustre: 63740:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1842.974391] Lustre: 63740:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1842.982358] Lustre: 63740:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1842.992066] Lustre: 63740:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1842.998948] Lustre: 63740:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1844.352754] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1853.460809] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.198639] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1899.489982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.512788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.831409] Lustre: server umount lustre-OST0000 complete [ 1907.913980] Lustre: server umount lustre-MDT0000 complete [ 1920.237318] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1923.996541] Key type lgssc unregistered [ 1924.463607] LNet: 65328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.477488] LNetError: 65328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.506418] LNet: Removed LNI 192.168.201.125@tcp [ 1925.746653] Key type .llcrypt unregistered [ 1925.748675] Key type ._llcrypt unregistered [ 1948.573495] Key type ._llcrypt registered [ 1948.575702] Key type .llcrypt registered [ 1948.737420] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 1964.755541] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1965.414358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1965.428731] alg: No test for adler32 (adler32-zlib) [ 1966.551310] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 1966.764025] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1968.415459] Key type lgssc registered [ 1969.738295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.408356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.461907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.861864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2009.932167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2010.128886] Lustre: lustre-MDT0000: new disk, initializing [ 2010.208415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.218821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2015.250286] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2026.935216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.040609] Lustre: 68488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2027.110312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2027.113530] Lustre: Skipped 1 previous similar message [ 2027.162695] Lustre: lustre-MDT0001: new disk, initializing [ 2027.257032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.317187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2027.329523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2031.948916] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2041.924978] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.401262] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.399360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.859134] Lustre: lustre-OST0000: new disk, initializing [ 2054.862274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2054.867629] Lustre: 70455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2054.987244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.789770] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2063.967580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2063.987153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2064.220016] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2071.778369] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2079.896538] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2080.335353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2087.148531] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2087.611758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2095.086910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.091841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2095.127737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.141922] LustreError: Skipped 1 previous similar message [ 2100.196627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.199393] Lustre: Skipped 1 previous similar message [ 2104.287125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2104.380347] Lustre: server umount lustre-OST0000 complete [ 2109.409760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2109.418357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.435726] Lustre: Skipped 1 previous similar message [ 2109.445339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.450359] Lustre: Skipped 1 previous similar message [ 2110.434084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.683477] Lustre: server umount lustre-MDT0000 complete [ 2117.042572] LustreError: 68478:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786306340 with bad export cookie 14573613606726846597 [ 2117.049255] LustreError: 68478:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2117.051646] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.363847] Lustre: server umount lustre-MDT0001 complete [ 2126.149363] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 16:12:27 (1786306347) [ 2139.355838] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2185.095669] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 2196.811226] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2239.474469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.763399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2239.801986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2239.877349] Lustre: lustre-MDT0000: new disk, initializing [ 2239.919359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.955328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2245.531059] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2256.744867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.885248] Lustre: 76480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2256.982170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2256.991796] Lustre: Skipped 1 previous similar message [ 2257.109759] Lustre: lustre-MDT0001: new disk, initializing [ 2257.263570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2257.299948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2257.311614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2263.351521] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2274.711180] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2281.736170] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.656676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.859482] Lustre: lustre-OST0000: new disk, initializing [ 2288.862041] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2288.867849] Lustre: 78450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2288.956826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.879733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2294.885181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2295.007581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2295.186398] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2304.895664] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2309.782822] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2310.005408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2315.093806] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2315.343399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2320.867405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2320.870507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.874962] LustreError: Skipped 1 previous similar message [ 2320.880152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.880162] Lustre: Skipped 2 previous similar messages [ 2320.925775] Lustre: Skipped 1 previous similar message [ 2323.664427] Lustre: server umount lustre-OST0000 complete [ 2329.061758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2329.073744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.133320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.145191] Lustre: Skipped 1 previous similar message [ 2341.347325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2341.358659] Lustre: Skipped 5 previous similar messages [ 2342.367541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2342.762966] Lustre: server umount lustre-MDT0000 complete [ 2346.479987] LustreError: 76490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.488501] LustreError: 76490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2347.062417] LustreError: 76473:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786306570 with bad export cookie 14573613606726847479 [ 2347.077656] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.082275] LustreError: 76473:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2347.825973] Lustre: server umount lustre-MDT0001 complete [ 2357.995480] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 16:16:19 (1786306579) [ 2365.322826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.949964] LustreError: 80863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.036082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.047797] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.087556] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2371.268740] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2376.181848] LustreError: 80863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.835853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.651764] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2393.808761] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2399.685612] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2406.373879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2406.720161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.725343] Lustre: Skipped 1 previous similar message [ 2413.205623] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2421.148435] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2424.324477] Lustre: 83536:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting aa8f2fb3-bd00-4e10-a98f-08ac8aeb8734 at adminstrative request [ 2426.149546] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.173087] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2436.586734] Lustre: MGS: Client aa8f2fb3-bd00-4e10-a98f-08ac8aeb8734 (at 0@lo) reconnecting [ 2436.593957] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xca3fe05c6ac0cf69 to 0xca3fe05c6ac0d209 [ 2436.604970] Lustre: MGC192.168.201.125@tcp: Connection restored to 0@lo (at 0@lo) [ 2462.705415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2462.864913] Lustre: lustre-OST0001: new disk, initializing [ 2462.872948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2462.876396] Lustre: 83928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2469.009870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2469.014138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2469.115776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2469.379979] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2478.710842] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2484.729909] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.732869] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2484.735394] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2484.747562] Lustre: Skipped 2 previous similar messages [ 2484.775775] Lustre: Skipped 1 previous similar message [ 2488.345831] Lustre: server umount lustre-OST0001 complete [ 2489.830977] LustreError: 82684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.849912] LustreError: 82684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2492.911834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.074259] LustreError: 82684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.093313] LustreError: 82684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2507.231327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2507.538357] Lustre: server umount lustre-OST0000 complete [ 2514.401155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2514.405681] LustreError: Skipped 1 previous similar message [ 2514.408883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.421224] Lustre: Skipped 2 previous similar messages [ 2517.886382] Lustre: server umount lustre-MDT0000 complete [ 2520.547827] LustreError: 80863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.570282] LustreError: 80863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2521.906822] LustreError: 80845:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786306745 with bad export cookie 14573613606726849033 [ 2521.925305] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.925592] LustreError: 80845:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2522.279886] Lustre: server umount lustre-MDT0001 complete [ 2533.779358] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 16:19:14 (1786306754) [ 2536.375151] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2538.365457] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 16:19:20 (1786306760) [ 2540.290274] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2542.661554] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 16:19:23 (1786306763) [ 2545.312727] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2547.419970] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 16:19:28 (1786306768) [ 2549.298476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2551.655848] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 16:19:32 (1786306772) [ 2553.675663] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2560.768949] loop: module loaded [ 2560.797318] loop0: detected capacity change from 0 to 4294967296 [ 2560.848725] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2562.993854] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2566.639995] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2571.232225] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2601.166456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.222811] LustreError: 88053:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2601.231063] LustreError: 88053:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2601.242557] LustreError: 88053:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2601.260891] LustreError: 88053:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2624.488160] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 2636.380630] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2675.144228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.562626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2675.586558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2675.735606] Lustre: lustre-MDT0000: new disk, initializing [ 2675.852377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.858797] Lustre: Skipped 1 previous similar message [ 2675.871376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2680.792943] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2692.398523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.554142] Lustre: 90922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2692.628070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2692.633867] Lustre: Skipped 11 previous similar messages [ 2692.948196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2697.546337] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2710.409225] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.670763] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2726.662496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.100058] Lustre: 92891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2735.308554] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2735.698726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2735.798732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2745.683668] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2750.825610] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2751.025429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2756.299579] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2756.579507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2761.712899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2761.716937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.722777] LustreError: Skipped 1 previous similar message [ 2761.723924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.732908] Lustre: Skipped 2 previous similar messages [ 2765.072943] Lustre: server umount lustre-OST0000 complete [ 2775.445897] Lustre: server umount lustre-MDT0000 complete [ 2777.066870] LustreError: 90933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.090986] LustreError: 90933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2779.467825] LustreError: 90913:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307002 with bad export cookie 14573613606726849649 [ 2779.471260] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.474841] LustreError: 90913:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2779.903938] Lustre: server umount lustre-MDT0001 complete [ 2790.813835] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 16:23:32 (1786307012) [ 2798.642263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.883692] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2816.964415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.256177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2817.268311] Lustre: Skipped 3 previous similar messages [ 2822.827993] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2832.763203] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2840.521875] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2849.765837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.803285] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2865.594864] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2871.464654] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2871.666536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2877.299415] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2877.635880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2885.543431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.802881] Lustre: lustre-OST0001: new disk, initializing [ 2885.810479] Lustre: Skipped 2 previous similar messages [ 2885.814087] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2885.821495] Lustre: Skipped 3 previous similar messages [ 2885.828716] Lustre: 98685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2893.464632] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2895.486740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2895.494122] Lustre: Skipped 2 previous similar messages [ 2895.506189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2895.558167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2903.642219] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2908.475860] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2908.716733] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2914.031581] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2914.254850] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2935.322574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2970.071054] Lustre: Failing over lustre-MDT0000 [ 2970.551385] Lustre: server umount lustre-MDT0000 complete [ 2971.106592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2971.114498] LustreError: Skipped 1 previous similar message [ 2971.119963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.134517] Lustre: Skipped 2 previous similar messages [ 2971.144940] LustreError: 95303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.174224] LustreError: 95303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2989.023163] Lustre: 66710:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786307195/real 1786307195] req@00000000bd0660db x1873077798724864/t0(0) o400->MGC192.168.201.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786307211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2989.069886] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.449423] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2997.452641] LDISKFS-fs (dm-0): recovery complete [ 2997.462807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.758309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3003.497694] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3003.503280] Lustre: 100082:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3004.919257] Lustre: 96031:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3004.935473] Lustre: 96031:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 3004.951676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3005.047492] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3005.383198] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3005.724833] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3005.741106] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3005.806796] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3006.063137] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3006.083719] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3006.110638] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3006.121929] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 3006.796905] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3006.799978] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3007.160332] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3007.172639] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3008.162364] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3008.176853] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 5 previous similar messages [ 3008.817355] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3008.839983] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 3009.161511] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3009.171346] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 3012.337765] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3012.353038] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 3013.005530] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3013.023938] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3013.351911] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3013.363915] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3020.626560] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3020.647543] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 25 previous similar messages [ 3021.290972] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3021.302685] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 25 previous similar messages [ 3021.624337] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3021.637532] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 25 previous similar messages [ 3036.667210] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3036.688447] Lustre: 101539:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 49 previous similar messages [ 3037.327443] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3037.348277] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3037.679349] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3037.692707] LustreError: 101539:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3063.007190] Lustre: 101539:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3063.015621] Lustre: 101539:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3063.033388] Lustre: 101539:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 35827587-95b2-438b-b79b-2d84d7ba9be0@192.168.201.25@tcp [ 3063.057644] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3063.081605] Lustre: 101539:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000e289ba09 x1873077782779520/t0(8589934778) o35->35827587-95b2-438b-b79b-2d84d7ba9be0@192.168.201.25@tcp:701/0 lens 392/0 e 0 to 0 dl 1786307296 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3063.137131] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3063.151549] Lustre: lustre-MDT0000: Denying connection for new client 35827587-95b2-438b-b79b-2d84d7ba9be0 (at 192.168.201.25@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3063.179286] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3063.238079] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3063.243327] Lustre: Skipped 2 previous similar messages [ 3063.263759] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3063.344522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3063.353779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3084.781966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3084.792515] Lustre: Skipped 4 previous similar messages [ 3090.307784] Lustre: server umount lustre-OST0000 complete [ 3100.977390] Lustre: server umount lustre-OST0001 complete [ 3107.296738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.317265] Lustre: Skipped 8 previous similar messages [ 3112.220155] Lustre: server umount lustre-MDT0000 complete [ 3112.426892] LustreError: 95304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.463019] LustreError: 95304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 3116.880812] LustreError: 98684:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307340 with bad export cookie 14573613606726900868 [ 3116.900185] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3117.558513] Lustre: server umount lustre-MDT0001 complete [ 3129.469613] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 16:29:10 (1786307350) [ 3138.510823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3138.516839] Lustre: *** cfs_fail_loc=197, val=0*** [ 3154.079803] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3159.199531] LustreError: 103516:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 3164.347571] Lustre: *** cfs_fail_loc=197, val=0*** [ 3164.353393] Lustre: Skipped 65 previous similar messages [ 3184.927370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.932507] Lustre: Skipped 3 previous similar messages [ 3191.478247] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3199.735787] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3202.477832] Lustre: server umount lustre-OST0000 complete [ 3212.191895] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 16:30:33 (1786307433) [ 3234.186288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.303690] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3250.943609] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.078724] LustreError: 105177:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3256.095511] LustreError: 105177:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3256.106586] LustreError: 105177:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3256.604070] Lustre: server umount lustre-MDT0000 complete [ 3297.130669] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3309.341313] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3351.519252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.830534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3351.861907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3351.919725] Lustre: lustre-MDT0000: new disk, initializing [ 3352.049875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3356.371271] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3369.184605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.372950] Lustre: 109354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.867811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3375.973987] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3387.783263] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3393.952548] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.345805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.621404] Lustre: 111323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3409.944713] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3410.236441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3421.592328] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3427.610546] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3427.826649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3433.077674] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3433.262808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3436.004404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.008191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.011501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.011511] Lustre: Skipped 8 previous similar messages [ 3436.025065] LustreError: Skipped 4 previous similar messages [ 3436.056675] Lustre: Skipped 1 previous similar message [ 3449.315190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3449.501776] Lustre: server umount lustre-OST0000 complete [ 3467.231250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3467.501194] Lustre: server umount lustre-MDT0000 complete [ 3471.842654] LustreError: 109363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.860437] LustreError: 109363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3472.267183] LustreError: 111322:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307695 with bad export cookie 14573613606726911536 [ 3472.273758] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.280897] LustreError: 111322:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3472.784993] Lustre: server umount lustre-MDT0001 complete [ 3482.492641] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 16:35:04 (1786307704) [ 3492.708530] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3495.722142] Key type lgssc unregistered [ 3496.016049] LNet: 113815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3496.030643] LNetError: 113815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3496.052417] LNet: Removed LNI 192.168.201.125@tcp [ 3497.164278] Key type .llcrypt unregistered [ 3497.166463] Key type ._llcrypt unregistered [ 3515.478723] Key type ._llcrypt registered [ 3515.481567] Key type .llcrypt registered [ 3515.618953] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3534.292346] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3535.813092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3535.834438] alg: No test for adler32 (adler32-zlib) [ 3536.901954] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 3537.118902] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3538.719247] Key type lgssc registered [ 3539.930360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3589.045327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3589.079350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.399886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3590.425955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3590.553265] Lustre: lustre-MDT0000: new disk, initializing [ 3590.673429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.697853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3596.288767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3608.593996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.893426] Lustre: lustre-OST0000: new disk, initializing [ 3608.897515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3608.901616] Lustre: Skipped 1 previous similar message [ 3608.909405] Lustre: 117992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3608.991295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.403792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3615.422171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3615.484966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3616.958791] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3628.709567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3629.003485] Lustre: lustre-OST0001: new disk, initializing [ 3629.018632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3629.039370] Lustre: 118857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3629.166407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3635.176750] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3637.405385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3637.414097] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3637.441318] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3647.980206] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.983846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.995093] Lustre: Skipped 1 previous similar message [ 3648.007854] Lustre: Skipped 1 previous similar message [ 3651.356567] Lustre: server umount lustre-MDT0000 complete [ 3658.720802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3663.463698] LustreError: 117179:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307886 with bad export cookie 4015351080749337184 [ 3663.478148] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.482436] LustreError: 117179:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3663.575944] Lustre: server umount lustre-OST0000 complete [ 3667.878336] Lustre: server umount lustre-OST0001 complete [ 3678.325931] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 16:38:19 (1786307899) [ 3704.258824] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 16:38:45 (1786307925) [ 3714.659604] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3738.624396] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 16:39:20 (1786307960) [ 3778.038822] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3788.288904] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3828.922213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.182634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3829.235677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.373739] Lustre: lustre-MDT0000: new disk, initializing [ 3829.537763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.570320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3835.226925] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3847.091854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.247850] Lustre: 126611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3847.327811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3847.338519] Lustre: Skipped 1 previous similar message [ 3847.467565] Lustre: lustre-MDT0001: new disk, initializing [ 3847.633207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.708102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3847.737734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3853.929274] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3865.940112] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3871.746540] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3880.279404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3880.552283] Lustre: lustre-OST0000: new disk, initializing [ 3880.555943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3880.561387] Lustre: 128582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3880.690213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.184699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3888.207640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3888.242751] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3888.352501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3900.126906] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.089788] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3906.423917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3910.889586] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3911.324352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3914.226980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.230108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3914.236421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.242198] Lustre: Skipped 1 previous similar message [ 3914.256686] LustreError: Skipped 1 previous similar message [ 3919.342295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3919.353666] Lustre: Skipped 2 previous similar messages [ 3919.853961] Lustre: server umount lustre-OST0000 complete [ 3929.568887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3929.573245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.585850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3929.617113] Lustre: Skipped 1 previous similar message [ 3932.000241] Lustre: server umount lustre-MDT0000 complete [ 3936.860449] LustreError: 127405:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308160 with bad export cookie 4015351080749338416 [ 3936.867532] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3936.873911] LustreError: 127405:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3937.652232] Lustre: server umount lustre-MDT0001 complete [ 3946.597375] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 16:42:48 (1786308168) [ 3953.494660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.948735] LustreError: 130996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.970243] LustreError: 130996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3954.034726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.274504] LustreError: 130997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.300939] LustreError: 130997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3959.467963] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3964.389875] LustreError: 130996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.514118] LustreError: 130997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.565208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.860513] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3985.582243] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3991.342142] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3998.380873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3998.713282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3998.715854] Lustre: Skipped 1 previous similar message [ 4004.770413] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4013.261143] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4021.574232] Lustre: *** cfs_fail_loc=159, val=0*** [ 4037.318471] Lustre: lustre-MDT0000: Client 672e9f15-500e-4a19-bbfb-871edc9e3b58 (at 192.168.201.25@tcp) reconnecting [ 4039.449472] Lustre: *** cfs_fail_loc=159, val=0*** [ 4039.451161] Lustre: Skipped 5 previous similar messages [ 4055.220372] Lustre: lustre-MDT0000: Client 672e9f15-500e-4a19-bbfb-871edc9e3b58 (at 192.168.201.25@tcp) reconnecting [ 4061.153030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4061.168106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.186575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4061.195946] Lustre: Skipped 1 previous similar message [ 4062.178172] LustreError: 132819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.199566] LustreError: 132819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4062.361541] Lustre: server umount lustre-OST0000 complete [ 4067.303138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.320647] Lustre: Skipped 2 previous similar messages [ 4072.420884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.434521] Lustre: Skipped 3 previous similar messages [ 4072.833637] Lustre: server umount lustre-MDT0000 complete [ 4076.947121] LustreError: 130977:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308300 with bad export cookie 4015351080749339298 [ 4076.962969] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4077.338895] Lustre: server umount lustre-MDT0001 complete [ 4087.452245] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4089.981900] Key type lgssc unregistered [ 4090.244031] LNet: 134776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4090.256340] LNetError: 134776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4091.301640] LNet: Removed LNI 192.168.201.125@tcp [ 4092.117202] Key type .llcrypt unregistered [ 4092.119447] Key type ._llcrypt unregistered [ 4121.106789] Key type ._llcrypt registered [ 4121.111630] Key type .llcrypt registered [ 4121.204704] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4122.053507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.124666] alg: No test for adler32 (adler32-zlib) [ 4123.342722] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 4123.698887] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4125.455905] Key type lgssc registered [ 4126.783323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4136.227556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4136.249408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.818828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.984253] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4147.174227] LustreError: 136193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.189969] LustreError: 136193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4152.287920] LustreError: 136192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.663250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.240777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4158.640211] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4166.583425] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4173.575814] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4180.765588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4181.156341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.186445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4186.786166] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4194.301248] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4206.502439] Lustre: *** cfs_fail_loc=159, val=0*** [ 4222.125465] Lustre: lustre-MDT0000: Client 3f12300f-efe6-41a0-a21a-e0980fc89583 (at 192.168.201.25@tcp) reconnecting [ 4225.180192] Lustre: *** cfs_fail_loc=159, val=0*** [ 4241.565303] Lustre: lustre-MDT0001: Client 3f12300f-efe6-41a0-a21a-e0980fc89583 (at 192.168.201.25@tcp) reconnecting [ 4243.358453] Lustre: *** cfs_fail_loc=159, val=0*** [ 4243.371503] Lustre: Skipped 4 previous similar messages [ 4258.993872] Lustre: lustre-MDT0001: Client 3f12300f-efe6-41a0-a21a-e0980fc89583 (at 192.168.201.25@tcp) reconnecting [ 4264.263772] Lustre: 139231:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4266.937525] Lustre: *** cfs_fail_loc=159, val=0*** [ 4266.946454] Lustre: Skipped 1 previous similar message [ 4283.563751] Lustre: lustre-MDT0000: Client 6e6de3e5-1eea-4148-be7f-b6636c4530c1 (at 192.168.201.25@tcp) reconnecting [ 4285.413164] Lustre: *** cfs_fail_loc=159, val=0*** [ 4285.430783] Lustre: Skipped 14 previous similar messages [ 4300.939609] Lustre: lustre-MDT0000: Client 6e6de3e5-1eea-4148-be7f-b6636c4530c1 (at 192.168.201.25@tcp) reconnecting [ 4302.935511] Lustre: 139524:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4307.424499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.428960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.436462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.445232] Lustre: Skipped 1 previous similar message [ 4312.545790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.548448] Lustre: Skipped 2 previous similar messages [ 4313.639612] Lustre: server umount lustre-OST0000 complete [ 4322.273798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4322.298671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.321275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.201777] Lustre: server umount lustre-MDT0000 complete [ 4327.905764] LustreError: 137158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.935671] LustreError: 137158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4328.263442] LustreError: 136171:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308551 with bad export cookie 4112555067159271639 [ 4328.270436] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4328.273362] LustreError: 136171:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4328.611457] Lustre: server umount lustre-MDT0001 complete [ 4341.108506] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4345.030612] Key type lgssc unregistered [ 4345.584763] LNet: 140580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4345.600608] LNetError: 140580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4346.666237] LNet: Removed LNI 192.168.201.125@tcp [ 4347.811231] Key type .llcrypt unregistered [ 4347.818348] Key type ._llcrypt unregistered [ 4379.323234] Key type ._llcrypt registered [ 4379.326568] Key type .llcrypt registered [ 4379.543502] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4380.976083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4381.134387] alg: No test for adler32 (adler32-zlib) [ 4382.578706] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 4383.043039] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4384.799253] Key type lgssc registered [ 4386.643348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4396.431718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4396.453138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.983416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.070796] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4408.294487] LustreError: 141998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.331655] LustreError: 141998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4411.996074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.485875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.763732] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4426.011935] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4431.409891] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4438.461974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.880959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.997484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4444.904040] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4454.088881] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4457.366133] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4463.279867] Lustre: 144583:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4466.811628] Lustre: 144632:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4468.590112] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4469.395574] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4473.680122] Lustre: 144778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4474.654672] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4475.917920] Lustre: 144886:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4483.049558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4483.073952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.108565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.119399] Lustre: Skipped 1 previous similar message [ 4483.131045] Lustre: Skipped 1 previous similar message [ 4487.122518] Lustre: server umount lustre-OST0000 complete [ 4493.288381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.293724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.309299] Lustre: Skipped 1 previous similar message [ 4497.483377] Lustre: server umount lustre-MDT0000 complete [ 4498.404310] LustreError: 141997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.431877] LustreError: 141997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4501.552044] LustreError: 141979:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308724 with bad export cookie 16777650458373954336 [ 4501.571305] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.948751] Lustre: server umount lustre-MDT0001 complete [ 4514.747812] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4518.219337] Key type lgssc unregistered [ 4518.539158] LNet: 145940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4518.555499] LNetError: 145940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4518.566716] LNet: Removed LNI 192.168.201.125@tcp [ 4519.571285] Key type .llcrypt unregistered [ 4519.578888] Key type ._llcrypt unregistered [ 4545.557424] Key type ._llcrypt registered [ 4545.563347] Key type .llcrypt registered [ 4545.723820] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4547.004496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4547.044174] alg: No test for adler32 (adler32-zlib) [ 4548.391487] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 4548.669553] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4550.503445] Key type lgssc registered [ 4552.239618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4562.307600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4562.315565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.750957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.734694] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4573.153561] LustreError: 147355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.161915] LustreError: 147355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4577.355683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.905194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.727942] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4589.940472] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4595.998455] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4605.543815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4605.938449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.027268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4613.817968] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4622.419263] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4622.563453] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4629.992591] Lustre: *** cfs_fail_loc=159, val=0*** [ 4646.542694] Lustre: lustre-MDT0000: Client 48a7471d-6caa-4cad-bb9a-8d80080c56c5 (at 192.168.201.25@tcp) reconnecting [ 4653.027216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4653.029817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4653.037869] Lustre: Skipped 1 previous similar message [ 4658.146052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4658.151133] Lustre: Skipped 1 previous similar message [ 4659.269615] Lustre: server umount lustre-OST0000 complete [ 4664.800342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4664.819231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.834440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.385486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.386138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.402890] Lustre: Skipped 1 previous similar message [ 4670.679162] Lustre: server umount lustre-MDT0000 complete [ 4673.507080] LustreError: 148385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.524976] LustreError: 148385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4674.894951] LustreError: 147336:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308898 with bad export cookie 17850427124814189526 [ 4674.910333] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4674.914903] LustreError: 147336:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4675.329816] Lustre: server umount lustre-MDT0001 complete [ 4689.640708] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4692.964208] Key type lgssc unregistered [ 4693.342153] LNet: 151026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4693.347199] LNetError: 151026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4694.381520] LNet: Removed LNI 192.168.201.125@tcp [ 4695.464249] Key type .llcrypt unregistered [ 4695.470554] Key type ._llcrypt unregistered [ 4725.913565] Key type ._llcrypt registered [ 4725.920424] Key type .llcrypt registered [ 4726.029172] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4727.333313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4727.443276] alg: No test for adler32 (adler32-zlib) [ 4728.761502] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 4729.190378] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4730.919397] Key type lgssc registered [ 4732.029945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4741.145714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4741.159715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.638499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.622678] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4751.852313] LustreError: 152442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.878860] LustreError: 152442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4755.912091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.163057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4762.117197] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4772.882246] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4780.832261] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4788.372796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4788.752156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.674605] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4801.116489] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4805.814841] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4812.480429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4832.016514] Lustre: lustre-OST0000: evicting 06fb4a30-97bc-4af4-880e-32a6d36860d7 (at 192.168.201.25@tcp) by administrative request [ 4832.024494] Lustre: Skipped 1 previous similar message [ 4862.432331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4862.435789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.475414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4863.980810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.984353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4863.984721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.012281] Lustre: Skipped 1 previous similar message [ 4869.088064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4869.109422] Lustre: Skipped 1 previous similar message [ 4874.210547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4874.224854] Lustre: Skipped 1 previous similar message [ 4875.232130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4875.529420] Lustre: server umount lustre-OST0000 complete [ 4884.449687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4884.451922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.493630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4884.502309] Lustre: Skipped 1 previous similar message [ 4886.425778] Lustre: server umount lustre-MDT0000 complete [ 4889.574334] LustreError: 152442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.603232] LustreError: 152442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4890.548458] LustreError: 152423:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309113 with bad export cookie 2204709289842574586 [ 4890.556353] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4890.815786] Lustre: server umount lustre-MDT0001 complete [ 4902.895592] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4905.730239] Key type lgssc unregistered [ 4905.994657] LNet: 156517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4906.001939] LNetError: 156517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4906.015651] LNet: Removed LNI 192.168.201.125@tcp [ 4907.098587] Key type .llcrypt unregistered [ 4907.103417] Key type ._llcrypt unregistered [ 4976.423721] Key type ._llcrypt registered [ 4976.426885] Key type .llcrypt registered [ 4976.549610] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4978.392159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.818877] alg: No test for adler32 (adler32-zlib) [ 4980.032665] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 4980.379529] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4982.095206] Key type lgssc registered [ 4984.357648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4994.456364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4994.467924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.057437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.855693] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5006.306611] LustreError: 159009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.330461] LustreError: 159009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5010.892284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.441597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5015.530921] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5022.642400] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5027.697983] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5034.698568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5034.984794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.029720] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5041.155761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5041.684263] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5049.861037] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5061.610486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5061.617537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.625188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5061.640497] Lustre: Skipped 1 previous similar message [ 5061.653596] Lustre: Skipped 1 previous similar message [ 5065.682209] Lustre: server umount lustre-OST0000 complete [ 5071.844233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.847746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.863246] Lustre: Skipped 1 previous similar message [ 5075.958234] Lustre: server umount lustre-MDT0000 complete [ 5076.963427] LustreError: 159009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.996733] LustreError: 159009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5080.049286] LustreError: 158991:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309303 with bad export cookie 1158948034665805749 [ 5080.054202] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5080.062658] LustreError: 158991:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5080.402588] Lustre: server umount lustre-MDT0001 complete [ 5091.633269] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5094.795964] Key type lgssc unregistered [ 5095.178115] LNet: 162566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5095.189181] LNetError: 162566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5095.212845] LNet: Removed LNI 192.168.201.125@tcp [ 5096.318971] Key type .llcrypt unregistered [ 5096.324069] Key type ._llcrypt unregistered [ 5121.749651] Key type ._llcrypt registered [ 5121.753955] Key type .llcrypt registered [ 5121.917807] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5122.815495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5122.832652] alg: No test for adler32 (adler32-zlib) [ 5123.944344] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 5124.258702] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5125.936783] Key type lgssc registered [ 5127.299267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5142.806392] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 17:02:44 (1786309364) [ 5162.120315] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5174.184343] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5216.564443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5216.595043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.010193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5218.094974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5218.270940] Lustre: lustre-MDT0000: new disk, initializing [ 5218.398474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.436483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5223.623739] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5236.049141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.195189] Lustre: 166832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5236.276288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5236.279633] Lustre: Skipped 1 previous similar message [ 5236.333525] Lustre: lustre-MDT0001: new disk, initializing [ 5236.484200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.534674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5236.550289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5241.792570] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5252.605141] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5259.853957] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5267.665166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5268.138974] Lustre: lustre-OST0000: new disk, initializing [ 5268.148488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5268.177552] Lustre: 168801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5268.340543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.275867] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5276.773326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5276.779675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5276.925117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5286.903967] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5297.639049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5297.640351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.642530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5297.653321] LustreError: Skipped 1 previous similar message [ 5297.681990] Lustre: Skipped 1 previous similar message [ 5302.758230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5302.769315] Lustre: Skipped 2 previous similar messages [ 5303.377764] Lustre: server umount lustre-OST0000 complete [ 5307.877605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.893018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.896152] Lustre: Skipped 1 previous similar message [ 5312.996301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5313.009098] Lustre: Skipped 1 previous similar message [ 5318.113697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5318.119348] Lustre: Skipped 1 previous similar message [ 5322.207392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5322.777914] Lustre: server umount lustre-MDT0000 complete [ 5326.828996] LustreError: 166825:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309549 with bad export cookie 7644912632046920284 [ 5326.835149] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.842690] LustreError: 166825:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5327.308148] Lustre: server umount lustre-MDT0001 complete [ 5339.851904] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5342.915626] Key type lgssc unregistered [ 5343.402487] LNet: 170713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5343.417556] LNetError: 170713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5343.436921] LNet: Removed LNI 192.168.201.125@tcp [ 5344.377473] Key type .llcrypt unregistered [ 5344.379448] Key type ._llcrypt unregistered [ 5368.089603] Key type ._llcrypt registered [ 5368.091631] Key type .llcrypt registered [ 5368.176064] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5369.827403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5369.857995] alg: No test for adler32 (adler32-zlib) [ 5371.094802] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 5371.440575] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5373.263177] Key type lgssc registered [ 5374.668931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5386.634569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5386.675585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.153746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.259664] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5398.507725] LustreError: 172134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.534025] LustreError: 172134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5403.452595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.621270] LustreError: 172135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.008423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5409.546720] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5418.103505] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.863226] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5434.280652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.583951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.589977] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5450.707825] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5458.120502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5467.323733] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5467.337459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.350352] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5467.356882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.368284] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5468.461118] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5468.490268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.503053] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5468.515901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.528874] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5469.795219] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5469.815563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.825741] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5469.830301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.840855] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5470.730127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.857911] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5471.861640] Lustre: Skipped 1 previous similar message [ 5471.881527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.888493] Lustre: Skipped 1 previous similar message [ 5471.892591] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5471.896418] Lustre: Skipped 1 previous similar message [ 5471.899137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.907502] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5471.912838] Lustre: Skipped 1 previous similar message [ 5472.832068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.750323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.596228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.344049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.426161] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5476.432542] Lustre: Skipped 4 previous similar messages [ 5476.458547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.469721] Lustre: Skipped 4 previous similar messages [ 5476.473880] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5476.481350] Lustre: Skipped 4 previous similar messages [ 5476.485043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.498796] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5476.503765] Lustre: Skipped 4 previous similar messages [ 5477.683912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.658964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.708409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.598925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.423292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.277265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.208052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.032613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.947199] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5484.952412] Lustre: Skipped 8 previous similar messages [ 5484.975059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.982392] Lustre: Skipped 8 previous similar messages [ 5484.989891] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5484.993318] Lustre: Skipped 8 previous similar messages [ 5484.996515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.012313] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5485.016168] Lustre: Skipped 8 previous similar messages [ 5485.782652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.698976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.732378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.819873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.714498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.696522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.658208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.564584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.559496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.721888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.526127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.488938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.378139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.448363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.335555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.401968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.345114] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5501.351625] Lustre: Skipped 16 previous similar messages [ 5501.376673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.388767] Lustre: Skipped 16 previous similar messages [ 5501.392634] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5501.399745] Lustre: Skipped 16 previous similar messages [ 5501.417099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.430346] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5501.439946] Lustre: Skipped 16 previous similar messages [ 5502.223663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.071819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.005903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.881193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.835414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.645997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.468797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.325442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.244791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.144680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.117786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.020729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.146841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.977874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.090910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.005892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.137790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.087857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.214486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.492237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.501892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.375823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.295530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.019628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.083812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.058407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.080570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.935273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.898877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.807618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.897474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.956436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.849316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.824087] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5533.830374] Lustre: Skipped 33 previous similar messages [ 5533.843723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.860968] Lustre: Skipped 33 previous similar messages [ 5533.865094] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5533.869261] Lustre: Skipped 33 previous similar messages [ 5533.875902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.884402] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5533.887141] Lustre: Skipped 33 previous similar messages [ 5534.908854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.967743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.945923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.746249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.886936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.806584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.829418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.745908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.913762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.928876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.994052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.187468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.263307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.306508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.265672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.232500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.213733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.076982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.087884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.960965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.949773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.753811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.841768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.816824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.008460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.289267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.061770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.164267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.948742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.896713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.922345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.117878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.205315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.153624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.182070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.168862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.358864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.416105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.524333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.370885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.631044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.755402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.964660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.932237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.117979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.085697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.223938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.331958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.189948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.186122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.323749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.390275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.375862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.479107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.482265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.494677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.381389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.465328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.371912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.296304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.270331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.376133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.310286] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5598.318910] Lustre: Skipped 62 previous similar messages [ 5598.334111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.344297] Lustre: Skipped 62 previous similar messages [ 5598.347741] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5598.355079] Lustre: Skipped 62 previous similar messages [ 5598.359270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.377241] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5598.379968] Lustre: Skipped 62 previous similar messages [ 5599.490373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.671988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.776467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.027306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.126856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.413815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.683719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.855719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.102205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.209295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.304705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.505816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.654291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.645741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.893568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.513563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.688262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.790550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.043378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.071206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.138749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.708939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.780791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.962360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.146572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.236970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.782045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.818795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.885453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.988978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.842270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.947760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.979679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.090731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.302159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.181887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.230085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.203145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.362398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.132441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.020556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.878148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.911449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.906989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.165349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.399721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.561408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.660490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.686191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.961629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.840331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.015547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.015805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.215845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.290794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.527540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.589144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.674742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.798820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.796828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.870817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.659816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.837502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.958125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.262815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.151408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.092607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.110540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.153388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.122927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.078046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.019381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.071767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.968437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.868873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.930642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.880151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.139461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.265768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.273933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.551975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.841070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.197393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.507558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.577634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.776219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.979907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.060055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.958675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.910157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.977386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.031717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.925729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.014047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.139192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.842409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.688942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.644618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.746273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.058564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.400234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.496574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.904083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.556883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.983966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.671051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.549419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.686015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.723883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.756225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.547858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.474414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.575562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.033189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.502895] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5727.518939] Lustre: Skipped 114 previous similar messages [ 5727.545987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.558728] Lustre: Skipped 114 previous similar messages [ 5727.562678] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5727.566724] Lustre: Skipped 114 previous similar messages [ 5727.575712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.643064] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5727.649403] Lustre: Skipped 114 previous similar messages [ 5729.198819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.335353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.762149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.821498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.149564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.496575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.046428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.091380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.312064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.749874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.908828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.927465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.062732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.308867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.583357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.537101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.821103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.251513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.430549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.590863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.760963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.554402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.656920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.769945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.883770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.724876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.704854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.589972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.510405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.550044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.563465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.545937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.513465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.426438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.516092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.494509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.600455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.640174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.893398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.767078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.742585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.582750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.624437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.739243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.762457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.924171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.998613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.131790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.242741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.330316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.303989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.565701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.559707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.415391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.402750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.356103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.229368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.328629] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5790.343098] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5790.356125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.429086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.523145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.214290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.032933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.969220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.140402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.167148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.463937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.541589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.527511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.681385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.776227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.723629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.846862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.807908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.869093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.723974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.829256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.782489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.664880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.711039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.601278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.591036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.553482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.542372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.654267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.842096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.886797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.230868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.126568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.285543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.308395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.240639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.170280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.131245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.224358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.291965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.365384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.431190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.365091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.488084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.306402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.409777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.317256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.529933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.602293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.699966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.759560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.794575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.781777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.832065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.994978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.078884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.011518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.096061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.933573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.956737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.131129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.135787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.209157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.406595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.496214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.425132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.355881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.448823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.444433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.637555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.678068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.741526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.794399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.844300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.927294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.101203] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5866.113383] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5866.135702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.098308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.179340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.159254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.181609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.306040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.509973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.563105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.716497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.742861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.707280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.865489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.030311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.901660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.991774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.939829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.010240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.117110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.142328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.029669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.189428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.031483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.947815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.948396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.994764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.133606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.212376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.489345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.650317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.714091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.835383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.790086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.998377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.092320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.220218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.216512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.455054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.336317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.365420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.238936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.182473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.079246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.189761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.137422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.186222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.139131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.455623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.439388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.687292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.954207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.083159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.158764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.226152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.536329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.745931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.004483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.059727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.054254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.411389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.756938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.852403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.935488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.849835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.885787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.880310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.944467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.962782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.260902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.463398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.709491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.740972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.695648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.669555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.818936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.782451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.694939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.662632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.923708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.085849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.167079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.352857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.395774] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5953.412508] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5953.419463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.871214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.011277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.991493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.142254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.208041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.152440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.335256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.249506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.250787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.197563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.200258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.110386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.084042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.120252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.059302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.026893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.934566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.844777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.866927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.057847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.200884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.320458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.359267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.241560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.086550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.021315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.825259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.746341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.592575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.492895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.249805] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5984.254499] Lustre: Skipped 242 previous similar messages [ 5984.269034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.278555] Lustre: Skipped 242 previous similar messages [ 5984.282895] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5984.290120] Lustre: Skipped 242 previous similar messages [ 5984.293920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.304859] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5984.318063] Lustre: Skipped 242 previous similar messages [ 5985.568179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.651677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.621200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.666823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.855789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.934897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.832703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.983268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.003851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.204138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.144123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.029726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.951970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.808437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.805806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.640267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.857248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.794108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.958201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.038827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.992585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.895248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.016431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.058624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.974663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.798540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.775468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.628469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.782089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.860631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.975287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.123497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.227742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.238525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.421750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.689490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.622399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.343501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.328357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.325074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.116148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.139724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.829891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.566661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.426598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.469149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.115572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.401183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.589274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.538122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.529673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.500350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.288672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.056188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.753917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.042541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.610326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.261653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.773984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.964578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.638416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.924585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.980950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.083344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.888902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.810791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.588851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.305490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.812467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.545538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.619287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.723325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.562234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.528085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.379841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.353510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.142307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.976467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.549298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.832683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.066789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.170769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.230035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.281410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.277227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.297211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.898095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.197972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.531592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.417734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.958283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.034839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.136202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.297819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.365529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.553513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.938082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.242457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.812719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.887858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.028533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.508651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.413070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.570702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.732083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.959559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.152523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.487939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.592391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.323648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.472920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.404387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.158574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.771095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.035533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.266534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.360337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.452425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.569492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.912678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.070864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.364881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.423062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.504269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.427158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.411077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.318032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.296206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.277148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.467537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.379511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.687338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.876446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.187449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.494294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.624393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.764455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.115564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.297713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.276540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.361485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.328737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.242589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.363213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.262195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.428467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.599270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.415922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.835922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.015842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.191850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.430519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.627121] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6149.641271] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6149.651421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.803435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.733442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.594783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.988729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.103701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.889690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.896619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.973301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.973890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.287611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.683116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.903581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.965963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.095289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.039690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.863969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.621194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.666315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.540637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.420902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.627374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.509695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.684805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.979885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.375708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.295679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.618039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.076211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.074420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.182175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.003250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.865381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.681558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.539892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.530687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.561271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.510272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.570210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.481799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.492311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.400380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.578949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.393513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.060292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.719122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.036968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.116039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.160065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.473955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.726561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.099295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.242604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.725988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.290845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.621652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.504271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.057758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.785991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.093993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.171182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.018857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.101978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.964714] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6218.982491] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6218.987210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.872445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.042620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.113260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.465552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.257481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.149325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.392988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.536945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.248899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.883752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.470704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.289794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.046093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.151136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.940856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.528068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.149467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.030372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.758761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.788922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.125761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.218089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.435914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.832158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.912309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.025080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.245866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.552285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.098668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.340072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.644522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.111176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.665053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.625085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.722464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.613711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.623464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.592197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.638645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.483452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.556724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.443559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.591840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.524828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.600031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.481794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.409983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.243739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.115334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.090913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.884851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.538488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.599205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.303304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.073177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.937294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.059233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.858800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.471599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.143101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.023746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.882463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.882563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.929783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.731840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.508241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.728983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.700834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.681648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.544978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.539252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.636883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.488565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.429585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.282156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.972914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.063920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.071789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.278041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.354176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.347032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.394062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.304451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.419067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.377935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.393524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.431766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.306979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.139678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.011402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.011272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.768236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.626562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.596247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.587545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.411430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.526270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.606985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.781769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.761438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.804353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.718130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.748397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.850056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.834984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.058606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.090255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.322733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.113107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.116600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.181976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.300128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.128496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.224168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.139055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.158742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.051399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.988115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.032607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.868320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.943495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.073276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.884932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.843499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.723759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.592555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.527457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.428365] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6344.442843] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6344.464099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.349514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.283347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.111680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.081928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.914982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.880661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.736942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.686325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.672326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.610268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.564897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.427924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.279712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.139833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.045964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.915166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.959795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.253262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.296364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.404274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.338983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.131447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.161057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.168725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.976682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.843981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.957184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.796834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.738922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.722903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.777593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.877315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.848092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.863717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.949725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.938129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.837989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.744227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.794339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.791496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.669954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.473559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.579883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.518153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.568881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.655623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.477688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.706854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.695905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.669475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.522957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.576301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.530139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.593455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.517112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.586411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.661213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.622594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.704856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.849650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.047817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.072636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.057973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.112209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.137706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.203333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.115811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.124582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.148598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.008673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.834147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.010645] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6415.021919] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6415.030110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.094696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.857661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.770979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.673619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.365983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.231790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.966669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.935496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.750622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.753829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.804988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.789907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.590667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.606753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.657296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.717687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.009376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.100212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.595600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.351817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.794253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.922960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.397753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.894632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.102733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.282665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.244875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.390022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.987350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.094461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.274519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.280156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.420893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.724576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.931055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.961437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.036685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.059690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.062949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.443104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.818914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.163291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.271496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.157267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.382831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.423155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.493408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.727450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.870032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.894114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.602462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.598317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.837952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.798552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.968445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.177459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.527681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.748301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.948103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.228175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.334141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.344346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.444326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.619679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.602384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.616987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.526251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.644426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.709995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.573839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.619380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.683677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.726482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.884634] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6496.891540] Lustre: Skipped 489 previous similar messages [ 6496.914879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.921525] Lustre: Skipped 489 previous similar messages [ 6496.924877] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6496.927381] Lustre: Skipped 489 previous similar messages [ 6496.930263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.939734] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6496.946674] Lustre: Skipped 489 previous similar messages [ 6497.924643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.891168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.771580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.729634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.501423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.635324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.767931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.733858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.779617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.027185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.100651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.229361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.178381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.320702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.295858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.477053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.615356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.842337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.973423] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6516.977714] LustreError: 172154:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6516.991689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6967.006552] LustreError: 173958:0:(ofd_dev.c:1787:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6967.036350] LustreError: 172154:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7710.177792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7710.183977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7710.196405] Lustre: Skipped 19 previous similar messages [ 7710.204291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7712.229685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7712.251500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.290730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.304428] Lustre: Skipped 1 previous similar message [ 7716.213630] Lustre: server umount lustre-OST0000 complete [ 7723.499134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7723.507146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7728.548474] Lustre: server umount lustre-MDT0000 complete [ 7728.627173] LustreError: 174577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7728.647513] LustreError: 174577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7733.734033] LustreError: 172135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.779065] LustreError: 172135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7733.823430] LustreError: 176336:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311956 with bad export cookie 3810787993914754609 [ 7733.826119] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7733.849090] LustreError: 176336:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7734.246884] Lustre: server umount lustre-MDT0001 complete [ 7747.218856] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 7751.104595] Key type lgssc unregistered [ 7751.451529] LNet: 227447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7751.460324] LNetError: 227447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7751.480009] LNet: Removed LNI 192.168.201.125@tcp [ 7752.939526] Key type .llcrypt unregistered [ 7752.942172] Key type ._llcrypt unregistered [ 7783.858107] Key type ._llcrypt registered [ 7783.864859] Key type .llcrypt registered [ 7784.120329] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 7786.097061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7786.499443] alg: No test for adler32 (adler32-zlib) [ 7788.084224] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 7788.455945] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 7790.303190] Key type lgssc registered [ 7792.239716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7803.386221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7803.418616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.902520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7810.492388] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7815.143931] LustreError: 228864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.156687] LustreError: 228864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7820.268645] LustreError: 228863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7821.000113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.422247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7826.206385] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7835.249214] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7843.261910] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7852.359536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7853.237529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.450183] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7871.508418] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7875.810163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7881.816856] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7884.429764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.456534] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7884.467079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7884.479871] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7884.485239] LustreError: 228883:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7887.331618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7887.338927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7887.353558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7888.515611] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 7891.431232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.436333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7893.556643] Lustre: server umount lustre-OST0000 complete [ 7904.496222] LustreError: 228883:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7930.962042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.424383] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7931.439788] Lustre: Skipped 1 previous similar message [ 7931.621643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.016821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7937.030415] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7940.860720] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 7941.099362] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7941.122738] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7950.641282] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7956.166358] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7963.485268] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff968b0670c000.ost_server_uuid 50 [ 7965.475266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff968b0670c000.ost_server_uuid in FULL state after 0 sec [ 7971.296437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7971.301785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7971.313132] LustreError: 232076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.643775] Lustre: server umount lustre-OST0000 complete [ 7980.003554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7980.008680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.024805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7983.531735] Lustre: server umount lustre-MDT0000 complete [ 7987.179186] LustreError: 228863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.222372] LustreError: 228863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7988.191794] Lustre: 228175:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312194/real 1786312194] req@000000003fafdf99 x1873083903346176/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786312210 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7988.210594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.221372] Lustre: Skipped 1 previous similar message [ 7988.314368] LustreError: 228844:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312211 with bad export cookie 6357175897469463050 [ 7988.328917] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7988.337302] LustreError: 228844:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7988.826774] Lustre: server umount lustre-MDT0001 complete [ 8002.401148] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 8007.144609] Key type lgssc unregistered [ 8007.606206] LNet: 234079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8007.617509] LNetError: 234079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8007.645777] LNet: Removed LNI 192.168.201.125@tcp [ 8009.184903] Key type .llcrypt unregistered [ 8009.186919] Key type ._llcrypt unregistered [ 8047.930946] Key type ._llcrypt registered [ 8047.933918] Key type .llcrypt registered [ 8048.103300] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8050.043873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8050.115091] alg: No test for adler32 (adler32-zlib) [ 8051.368394] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 8051.755746] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 8053.503172] Key type lgssc registered [ 8055.234323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8068.034643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8068.056422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.842615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.718599] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8079.394414] LustreError: 235810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8079.428917] LustreError: 235810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8084.450194] LustreError: 235811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.631515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.174054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8094.477371] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8097.735872] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8106.847614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.559111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8114.749726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8115.030740] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8127.793229] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.984292] Lustre: lustre-OST0001: new disk, initializing [ 8127.987209] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8127.998382] Lustre: 238120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8128.155811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8135.803294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8135.812673] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8135.965613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8136.222582] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8150.311166] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8157.337960] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8175.017052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.028367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.043846] Lustre: Skipped 3 previous similar messages [ 8175.064594] Lustre: Skipped 3 previous similar messages [ 8180.205816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8180.216151] Lustre: Skipped 3 previous similar messages [ 8181.388861] Lustre: server umount lustre-MDT0000 complete [ 8185.317294] LustreError: 235805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.335922] LustreError: 235805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8185.607069] LustreError: 235793:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312408 with bad export cookie 15533474216680135053 [ 8185.627893] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8185.636087] LustreError: 235793:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8186.074119] Lustre: server umount lustre-MDT0001 complete [ 8200.129607] Lustre: server umount lustre-OST0000 complete [ 8214.734824] Lustre: server umount lustre-OST0001 complete [ 8241.796730] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8254.613452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.141112] LustreError: 241234:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.256299] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.584517] LustreError: 241235:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.850445] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8270.823559] LustreError: 241234:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.852829] LustreError: 241234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8273.057856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.727105] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8279.567909] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8283.409938] Lustre: 242378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8292.309385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8293.123805] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.228331] LustreError: 242731:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.247259] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 8300.275873] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8300.600583] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8312.488324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8321.420181] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8330.220760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8333.724262] Lustre: 244249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8345.570505] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8345.576179] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.588457] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.154516] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.160628] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.180039] Lustre: Skipped 2 previous similar messages [ 8349.190098] Lustre: Skipped 3 previous similar messages [ 8351.704962] Lustre: server umount mylustre-MDT0000 complete [ 8354.275188] LustreError: 241969:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8354.283114] LustreError: 241969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8355.859457] LustreError: 241212:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312579 with bad export cookie 15533474216680137916 [ 8355.863882] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8355.865497] LustreError: 241212:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8356.453168] Lustre: server umount mylustre-MDT0001 complete [ 8370.089577] Lustre: server umount mylustre-OST0000 complete [ 8384.433633] Lustre: server umount mylustre-OST0001 complete [ 8415.224386] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8427.838409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.159967] LustreError: 246561:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8428.236228] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.246892] Lustre: Skipped 1 previous similar message [ 8433.546248] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8443.912091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.134642] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8453.523386] Lustre: 247704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8460.797015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8469.220521] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8469.494615] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:161) [ 8473.600672] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8480.295806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8485.938245] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8488.399779] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8497.128753] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8501.252483] Lustre: 249573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8511.459689] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8511.462721] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.487237] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.532453] Lustre: Skipped 2 previous similar messages [ 8517.563548] Lustre: server umount tfs-MDT0000 complete [ 8521.648593] LustreError: 246537:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312744 with bad export cookie 15533474216680139862 [ 8521.652479] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8521.663602] LustreError: 246537:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8521.705200] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.707544] Lustre: tfs-MDT0001: Not available for connect from 0@lo (stopping) [ 8521.722681] Lustre: Skipped 7 previous similar messages [ 8521.723078] Lustre: Skipped 2 previous similar messages [ 8522.066902] Lustre: server umount tfs-MDT0001 complete [ 8528.933666] Lustre: server umount tfs-OST0000 complete [ 8533.794451] Lustre: server umount tfs-OST0001 complete [ 8556.874705] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8569.044757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.645343] LustreError: 251823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.666660] LustreError: 251823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8569.783560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.796676] Lustre: Skipped 3 previous similar messages [ 8575.753641] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8585.715527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.991424] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8595.823164] Lustre: 252966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8603.411434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8610.485330] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8611.076838] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:193) [ 8614.230897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8620.293796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8625.675115] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8628.880327] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8637.063433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8641.535324] Lustre: 254850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8648.597130] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 18:01:09 (1786312869) [ 8666.610807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.627685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.642751] Lustre: Skipped 2 previous similar messages [ 8666.687560] Lustre: Skipped 4 previous similar messages [ 8672.751014] Lustre: server umount lustre-MDT0000 complete [ 8677.828561] LustreError: 252969:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312900 with bad export cookie 15533474216680141787 [ 8677.831093] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.838475] LustreError: 252969:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8692.191434] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8692.583132] Lustre: server umount lustre-MDT0001 complete [ 8711.135768] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8711.451039] Lustre: server umount lustre-OST0000 complete [ 8722.490691] Lustre: server umount lustre-OST0001 complete [ 8730.496869] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 8741.221051] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 8784.273424] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8784.273424] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8784.273424] [ 8784.329580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8784.716901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8784.782819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8784.947885] Lustre: lustre-MDT0000: new disk, initializing [ 8785.053299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8785.061770] Lustre: Skipped 3 previous similar messages [ 8785.100236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8790.972663] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8802.947966] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8802.947966] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8802.947966] [ 8802.971915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8803.104666] Lustre: 258044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8803.141686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8803.145457] Lustre: Skipped 1 previous similar message [ 8803.229444] Lustre: lustre-MDT0001: new disk, initializing [ 8803.366426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8803.389942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8808.259738] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8818.978397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8819.365618] Lustre: lustre-OST0000: new disk, initializing [ 8819.369185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8819.373648] Lustre: 259626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8821.275656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8821.289939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8821.431118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8826.095773] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8836.826991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8837.070738] Lustre: lustre-OST0001: new disk, initializing [ 8837.076691] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8837.088350] Lustre: 260496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8838.996428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8839.011209] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8839.138752] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8844.524468] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 8857.699815] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 18:04:38 (1786313078) [ 8876.278669] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 18:04:57 (1786313097) [ 8892.896591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8892.912504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8892.936995] Lustre: Skipped 3 previous similar messages [ 8892.949250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8892.963207] Lustre: Skipped 10 previous similar messages [ 8905.183572] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8905.305980] Lustre: server umount lustre-OST0000 complete [ 8905.733974] LustreError: 259609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.766837] LustreError: 259609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 8910.816357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8910.834165] LustreError: Skipped 1 previous similar message [ 8917.204777] Lustre: server umount lustre-MDT0000 complete [ 8922.485802] LustreError: 258035:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313145 with bad export cookie 15533474216680143565 [ 8922.505822] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8922.508844] LustreError: 258035:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8926.178774] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8926.219821] Lustre: Skipped 4 previous similar messages [ 8928.900377] Lustre: server umount lustre-MDT0001 complete [ 8941.459145] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9012.574767] Lustre: server umount lustre-OST0001 complete [ 9019.421769] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9036.136468] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9077.717540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.985688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.023636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9078.101884] Lustre: lustre-MDT0000: new disk, initializing [ 9078.196975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9078.199952] Lustre: Skipped 3 previous similar messages [ 9078.221216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9083.037504] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9094.217327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.364867] Lustre: 265301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9094.432568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9094.436392] Lustre: Skipped 1 previous similar message [ 9094.524554] Lustre: lustre-MDT0001: new disk, initializing [ 9094.727987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9094.752200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9099.713910] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9110.011376] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9116.851888] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9124.348526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.619580] Lustre: lustre-OST0000: new disk, initializing [ 9124.622833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9124.630744] Lustre: 267269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9126.171018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9126.177179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9126.310842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9131.235788] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9142.884660] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9151.969892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9151.978333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.981762] LustreError: Skipped 1 previous similar message [ 9151.993101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9152.007491] Lustre: Skipped 14 previous similar messages [ 9157.383375] Lustre: server umount lustre-OST0000 complete [ 9168.013032] Lustre: server umount lustre-MDT0000 complete [ 9172.677293] LustreError: 268280:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313395 with bad export cookie 15533474216680145028 [ 9172.690831] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9172.696758] LustreError: 268280:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.241113] Lustre: server umount lustre-MDT0001 complete [ 9184.898653] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9187.777366] Key type lgssc unregistered [ 9188.093736] LNet: 269035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9188.098705] LNetError: 269035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9188.113150] LNet: Removed LNI 192.168.201.125@tcp [ 9189.048199] Key type .llcrypt unregistered [ 9189.049835] Key type ._llcrypt unregistered [ 9218.363950] Key type ._llcrypt registered [ 9218.365376] Key type .llcrypt registered [ 9218.477737] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9219.621355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9219.729707] alg: No test for adler32 (adler32-zlib) [ 9221.170093] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 9221.569263] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9223.416048] Key type lgssc registered [ 9225.143063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9235.499825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9235.545103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9237.152200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9242.326793] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9247.712101] LustreError: 270498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.735125] LustreError: 270498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9252.515214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.851032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9258.763119] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9267.472295] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9273.277446] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9281.103817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9281.576960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9282.641569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9288.078470] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9296.814424] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.987516] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9301.984859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9301.987864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9301.991459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9302.011829] Lustre: Skipped 1 previous similar message [ 9302.028020] Lustre: Skipped 1 previous similar message [ 9306.603192] Lustre: server umount lustre-OST0000 complete [ 9312.224263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.241604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9312.254587] Lustre: Skipped 1 previous similar message [ 9316.723407] Lustre: server umount lustre-MDT0000 complete [ 9317.344433] LustreError: 270499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.355583] LustreError: 270499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9320.404952] LustreError: 270479:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313543 with bad export cookie 2192220177666627358 [ 9320.409453] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9320.413542] LustreError: 270479:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9321.038067] Lustre: server umount lustre-MDT0001 complete [ 9334.481375] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9338.249085] Key type lgssc unregistered [ 9338.591525] LNet: 274088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9338.598588] LNetError: 274088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9338.609852] LNet: Removed LNI 192.168.201.125@tcp [ 9339.753250] Key type .llcrypt unregistered [ 9339.757072] Key type ._llcrypt unregistered [ 9357.133899] Key type ._llcrypt registered [ 9357.135425] Key type .llcrypt registered [ 9357.215770] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9358.251060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9358.270800] alg: No test for adler32 (adler32-zlib) [ 9359.242801] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 9359.443330] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9361.104985] Key type lgssc registered [ 9362.308455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9371.549077] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9371.561354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.039889] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9392.160860] LustreError: 275283:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 9418.848582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.586500] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9433.771633] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9436.014327] Lustre: server umount lustre-OST0000 complete [ 9456.227916] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 9459.164420] Key type lgssc unregistered [ 9459.454906] LNet: 276919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9459.462481] LNetError: 276919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9459.479049] LNet: Removed LNI 192.168.201.125@tcp [ 9460.559722] Key type .llcrypt unregistered [ 9460.566653] Key type ._llcrypt unregistered [ 9503.735595] Key type ._llcrypt registered [ 9503.737804] Key type .llcrypt registered [ 9503.869431] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9504.923273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9505.004935] alg: No test for adler32 (adler32-zlib) [ 9506.117503] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 9506.380974] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 9508.056418] Key type lgssc registered [ 9509.315397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9524.907380] loop0: detected capacity change from 0 to 204800000 [ 9525.165722] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.178477] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.459331] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9526.674821] loop0: detected capacity change from 0 to 204800000 [ 9526.911701] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9526.929030] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9527.471792] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9528.712362] loop0: detected capacity change from 0 to 204800000 [ 9528.964650] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9528.976306] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9529.091779] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9530.168833] loop0: detected capacity change from 0 to 204800000 [ 9530.333653] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9530.342231] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9530.462963] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9531.630724] loop0: detected capacity change from 0 to 204800000 [ 9531.656793] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9536.600237] loop0: detected capacity change from 0 to 204800000 [ 9536.634614] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9541.134841] loop0: detected capacity change from 0 to 204800000 [ 9541.157780] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9544.942880] loop0: detected capacity change from 0 to 204800000 [ 9544.965336] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9548.989394] loop0: detected capacity change from 0 to 204800000 [ 9549.059452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9549.089346] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.155562] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9549.161759] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9549.169268] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9549.176828] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9549.186936] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9549.194615] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9549.205550] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9549.213932] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9549.226667] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9549.236908] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9549.245598] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9549.255786] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9549.265967] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9549.277565] Lustre: 280334:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9555.700283] Lustre: server umount MGS complete [ 9556.536240] loop0: detected capacity change from 0 to 204800000 [ 9556.581818] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.250360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.326574] Lustre: lustre-MDT0000: Aborting client recovery [ 9558.299236] loop1: detected capacity change from 0 to 204800000 [ 9558.443304] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.651254] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9558.670358] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9558.689096] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9558.724805] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9558.745921] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9558.750254] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9558.755650] Lustre: 280767:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9558.795396] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9558.942795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9558.983851] Lustre: lustre-MDT0001: Aborting client recovery [ 9559.938838] loop2: detected capacity change from 0 to 204800000 [ 9560.162058] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.291365] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9560.298369] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9560.305349] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9560.311372] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9560.320681] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9560.328121] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9560.335402] Lustre: 280846:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9560.509467] Lustre: Found index 0 for lustre-OST0000, updating log [ 9560.595546] Lustre: 280864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.701730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9560.739022] Lustre: lustre-OST0000: Aborting recovery [ 9561.834279] loop3: detected capacity change from 0 to 204800000 [ 9562.070554] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9562.162858] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9562.193854] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9562.212482] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9562.230394] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9562.258385] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9562.275614] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9562.290086] Lustre: 280929:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9562.366576] Lustre: Found index 1 for lustre-OST0001, updating log [ 9562.415842] Lustre: 280934:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9562.569865] Lustre: lustre-OST0001: Aborting recovery [ 9567.660077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9567.664088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9567.705625] LustreError: 280710:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9567.742160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9567.742523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9569.707591] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9569.718506] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9572.514536] LustreError: 280677:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 291us: rc = -2 [ 9573.039546] LustreError: 280678:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 717us: rc = -2 [ 9573.074996] LustreError: 280678:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 9589.727980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9589.746821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9589.763919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9590.270292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9590.287290] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.295601] Lustre: Skipped 1 previous similar message [ 9590.345221] Lustre: Skipped 2 previous similar messages [ 9593.057952] Lustre: server umount lustre-MDT0000 complete [ 9594.773792] LustreError: 280661:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313817 with bad export cookie 18223822809502384083 [ 9594.789838] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9595.361434] LustreError: 281169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.363412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.388109] LustreError: 281169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9595.421839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9595.435234] Lustre: Skipped 3 previous similar messages [ 9599.411841] LustreError: 280676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.411995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9599.472319] Lustre: Skipped 1 previous similar message [ 9604.524457] LustreError: 280676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9604.524626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9604.547763] LustreError: 280676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9609.183654] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9609.621825] Lustre: server umount lustre-MDT0001 complete [ 9625.055157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9625.164764] Lustre: server umount lustre-OST0000 complete [ 9632.505638] Lustre: server umount lustre-OST0001 complete [ 9640.836797] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 18:17:42 (1786313862) [ 9670.760889] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 9680.281499] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 9716.023973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.234917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9716.279130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9716.381338] Lustre: lustre-MDT0000: new disk, initializing [ 9716.480117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9716.486870] Lustre: Skipped 1 previous similar message [ 9716.496972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9720.717217] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9731.133310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.248651] Lustre: 285009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9731.290780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9731.296913] Lustre: Skipped 1 previous similar message [ 9731.383397] Lustre: lustre-MDT0001: new disk, initializing [ 9731.465455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9731.515551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9731.532489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9738.304389] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9748.686882] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9755.514982] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.301951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.816653] Lustre: lustre-OST0000: new disk, initializing [ 9763.820189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9763.825818] Lustre: 286979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9763.908745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9770.654919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9770.676933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9770.864582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9771.467022] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9783.458561] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9789.522903] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9800.697207] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9805.000542] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9815.277194] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9858.018178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9858.028187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9858.039268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9858.045791] Lustre: Skipped 2 previous similar messages [ 9858.059422] Lustre: Skipped 2 previous similar messages [ 9864.040418] Lustre: server umount lustre-OST0000 complete [ 9868.258485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.276215] Lustre: Skipped 1 previous similar message [ 9874.595684] Lustre: server umount lustre-MDT0000 complete [ 9878.508427] LustreError: 285021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9878.529156] LustreError: 285021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9878.808064] LustreError: 285000:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314101 with bad export cookie 18223822809502386799 [ 9878.815967] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9878.831339] LustreError: 285000:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9879.167803] Lustre: server umount lustre-MDT0001 complete [ 9888.237964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.264895] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9898.385635] Lustre: server umount MGS complete [ 9910.738224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.231709] LustreError: 290804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.246102] LustreError: 290817:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9911.307380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9916.375418] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9921.505440] LustreError: 290805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.544235] LustreError: 290805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9926.175778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9932.164862] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9940.349645] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9946.931228] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9954.356260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9954.757876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.762511] Lustre: Skipped 1 previous similar message [ 9961.219465] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 9968.849708] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10001.908622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.933160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10001.933626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10001.936393] LustreError: Skipped 1 previous similar message [10001.954858] Lustre: Skipped 7 previous similar messages [10005.886851] Lustre: server umount lustre-OST0000 complete [10016.997961] Lustre: server umount lustre-MDT0000 complete [10017.268063] LustreError: 291534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10017.303902] LustreError: 291534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10021.876653] LustreError: 292650:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314245 with bad export cookie 18223822809502388717 [10021.890205] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10021.891415] LustreError: 292650:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [10022.432562] Lustre: server umount lustre-MDT0001 complete [10033.708765] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [10036.847300] Key type lgssc unregistered [10037.168788] LNet: 294755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10037.172556] LNetError: 294755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10037.196322] LNet: Removed LNI 192.168.201.125@tcp [10038.117437] Key type .llcrypt unregistered [10038.119657] Key type ._llcrypt unregistered [10077.339266] Key type ._llcrypt registered [10077.340492] Key type .llcrypt registered [10077.408104] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [10092.425164] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [10094.066504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10094.206392] alg: No test for adler32 (adler32-zlib) [10095.231772] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [10095.473102] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [10097.159246] Key type lgssc registered [10098.639898] Lustre: Echo OBD driver; http://www.lustre.org/ [10137.354704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10137.393523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10138.670451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10138.682271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10138.732879] Lustre: lustre-MDT0000: new disk, initializing [10138.787390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10138.806782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10144.303485] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10158.608432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.782070] Lustre: 298715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10158.911758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10158.916772] Lustre: Skipped 1 previous similar message [10159.104290] Lustre: lustre-MDT0001: new disk, initializing [10159.270177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10159.368382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10159.388830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10165.424875] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10176.203237] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10182.869415] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10190.301675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10190.692889] Lustre: lustre-OST0000: new disk, initializing [10190.701862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10190.711557] Lustre: 300685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10190.843519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10197.838914] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10198.101391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10198.126022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10198.257490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10208.026944] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10214.917855] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10226.724875] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10232.392192] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10280.416396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10280.419941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10280.429891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10280.440486] Lustre: Skipped 1 previous similar message [10285.543133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10285.550658] Lustre: Skipped 1 previous similar message [10286.654522] Lustre: server umount lustre-OST0000 complete [10290.658725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10290.675769] Lustre: Skipped 1 previous similar message [10290.684180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.782231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.784180] Lustre: Skipped 2 previous similar messages [10296.761121] Lustre: server umount lustre-MDT0000 complete [10300.772461] LustreError: 300684:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314523 with bad export cookie 9053022700219345073 [10300.774419] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10300.792066] LustreError: 300684:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [10301.343975] Lustre: server umount lustre-MDT0001 complete [10313.341709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.843230] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10328.791937] Lustre: server umount MGS complete [10344.160528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.903576] LustreError: 304569:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10344.998991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.061279] LustreError: 304556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10350.091858] LustreError: 304556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10351.035891] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10355.205109] LustreError: 304557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.290462] LustreError: 304556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10361.493749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.044179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10368.693894] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10381.087351] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10389.581935] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10396.823446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10397.078970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10403.612700] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10410.959980] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10446.304804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10446.312854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.315709] LustreError: Skipped 1 previous similar message [10446.316651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10446.341673] Lustre: Skipped 2 previous similar messages [10452.675424] Lustre: server umount lustre-OST0000 complete [10459.618261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10459.628187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10459.651936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10459.667944] Lustre: Skipped 3 previous similar messages [10471.391173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10471.536941] Lustre: server umount lustre-MDT0000 complete [10471.903859] LustreError: 304556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.913975] LustreError: 304556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10475.495225] LustreError: 306401:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314698 with bad export cookie 9053022700219346914 [10475.498644] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10475.506340] LustreError: 306401:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [10475.810916] Lustre: server umount lustre-MDT0001 complete [10489.066464] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [10492.133877] Key type lgssc unregistered [10492.369935] LNet: 308507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10492.375229] LNetError: 308507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10493.416112] LNet: Removed LNI 192.168.201.125@tcp [10494.419539] Key type .llcrypt unregistered [10494.425645] Key type ._llcrypt unregistered [10517.782357] Key type ._llcrypt registered [10517.785125] Key type .llcrypt registered [10517.912719] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [10534.314552] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [10535.630025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10535.720414] alg: No test for adler32 (adler32-zlib) [10537.037158] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [10537.304075] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [10539.064691] Key type lgssc registered [10540.200655] Lustre: Echo OBD driver; http://www.lustre.org/ [10567.887944] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [10586.212819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10586.253199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10587.497728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10587.519641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10587.588874] Lustre: lustre-MDT0000: new disk, initializing [10587.704883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10587.738671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10592.855623] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10604.125696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10604.430298] Lustre: lustre-OST0000: new disk, initializing [10604.435370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10604.441405] Lustre: Skipped 1 previous similar message [10604.447232] Lustre: 312561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10604.511446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10611.651644] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [10611.805373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10611.815589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10611.844891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11539.425508] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.434774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11541.856912] Lustre: server umount lustre-MDT0000 complete [11545.930975] LustreError: 311747:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315769 with bad export cookie 4830437508781003242 [11545.942785] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11545.947249] LustreError: 311747:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [11546.113988] Lustre: server umount lustre-OST0000 complete [11576.473988] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [11587.575447] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [11631.344272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.594724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11631.653702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11631.804268] Lustre: lustre-MDT0000: new disk, initializing [11631.964937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11631.994734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11636.921605] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11648.400599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.508467] Lustre: 316912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11648.561740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11648.566037] Lustre: Skipped 1 previous similar message [11648.661448] Lustre: lustre-MDT0001: new disk, initializing [11648.787283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11648.834846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11648.842627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11654.031849] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11663.671792] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11670.075708] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11677.628473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11678.007875] Lustre: lustre-OST0000: new disk, initializing [11678.022178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11678.030419] Lustre: 318881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11678.183470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11684.495218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11684.514157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11684.638311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11685.375422] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11696.285250] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11701.794578] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11702.063695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11707.396651] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11707.704323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11710.435564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.440965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11710.446192] Lustre: Skipped 1 previous similar message [11715.554632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.569572] Lustre: Skipped 2 previous similar messages [11720.678072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11720.685262] Lustre: Skipped 1 previous similar message [11724.255168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11724.556443] Lustre: server umount lustre-OST0000 complete [11730.923656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.954256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11730.959564] Lustre: Skipped 1 previous similar message [11735.550305] Lustre: server umount lustre-MDT0000 complete [11739.496837] LustreError: 316905:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315962 with bad export cookie 4830437508783944061 [11739.508301] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11739.943403] Lustre: server umount lustre-MDT0001 complete [11743.904634] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11746.204572] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 18:52:47 (1786315967) [11753.355454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11754.168692] LustreError: 321198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.183177] LustreError: 321198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11754.241700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11759.619954] LustreError: 321199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.718369] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11764.708129] LustreError: 321198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.850791] LustreError: 321199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.876396] LustreError: 321199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11773.942593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11774.661301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11780.279695] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11788.767952] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11794.884345] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11801.929878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11802.251709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.740952] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11817.441163] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11824.556690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11824.667715] Lustre: lustre-OST0001: new disk, initializing [11824.674527] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11824.681871] Lustre: 324037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11824.789344] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11831.662885] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11833.445181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11833.453252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11833.497881] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11834.552031] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11834.577081] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11835.630685] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11835.642341] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11837.735207] LustreError: 324490:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11837.747255] LustreError: 324490:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11841.682697] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11841.839228] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11841.857070] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11841.998865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11849.040753] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff968b0b1b1800.ost_server_uuid 50 [11850.109740] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11850.129174] LustreError: 324499:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [11850.900189] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff968b0b1b1800.ost_server_uuid in FULL state after 0 sec [11857.061570] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff968b0b1b1800.ost_server_uuid 50 [11858.830053] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff968b0b1b1800.ost_server_uuid in FULL state after 0 sec [11860.438843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11874.792609] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.795754] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11874.814376] Lustre: Skipped 2 previous similar messages [11874.833362] Lustre: Skipped 1 previous similar message [11881.053945] Lustre: server umount lustre-OST0001 complete [11883.116361] LustreError: 323020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11888.610363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11888.629131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11893.277412] Lustre: server umount lustre-OST0000 complete [11897.312688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11897.318458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11897.330095] Lustre: Skipped 1 previous similar message [11897.333879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11897.338052] Lustre: Skipped 6 previous similar messages [11903.220649] Lustre: server umount lustre-MDT0000 complete [11905.506635] LustreError: 321195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11905.525703] LustreError: 321195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11906.515920] LustreError: 321181:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316129 with bad export cookie 4830437508783944943 [11906.517954] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11906.527803] LustreError: 321181:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 7 previous similar messages [11906.866701] Lustre: server umount lustre-MDT0001 complete [11916.307830] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [11919.004597] Key type lgssc unregistered [11919.359567] LNet: 326728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11919.366373] LNetError: 326728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11920.427253] LNet: Removed LNI 192.168.201.125@tcp [11921.260920] Key type .llcrypt unregistered [11921.262931] Key type ._llcrypt unregistered [11945.023440] Key type ._llcrypt registered [11945.026701] Key type .llcrypt registered [11945.126502] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [11946.716046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11946.753537] alg: No test for adler32 (adler32-zlib) [11947.909846] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [11948.198543] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [11949.871177] Key type lgssc registered [11950.964633] Lustre: Echo OBD driver; http://www.lustre.org/ [11957.854026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11957.868273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11959.216450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11962.626230] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11969.507550] LustreError: 328151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.544283] LustreError: 328151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11970.533177] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11974.626670] LustreError: 328150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.786541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11979.147635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11984.897662] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [11994.733630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11995.132895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12001.624453] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12004.339281] LustreError: 329832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.354181] LustreError: 329832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12004.355773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12009.245271] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12009.445764] LustreError: 329831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.458797] LustreError: 329831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12014.569834] LustreError: 329832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.590227] LustreError: 329832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12016.372527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12016.813833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12022.283264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12023.358623] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12031.280287] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12037.669279] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12037.916668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12145.120375] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12145.132962] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.158444] Lustre: Skipped 1 previous similar message [12145.160084] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12145.181938] Lustre: Skipped 1 previous similar message [12146.464223] Lustre: server umount lustre-OST0001 complete [12150.242763] LustreError: 330306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12150.251850] LustreError: 330306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12152.801249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12152.816077] LustreError: Skipped 1 previous similar message [12152.817427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.841219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12155.363513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12155.371633] Lustre: Skipped 2 previous similar messages [12160.482395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12160.484948] Lustre: Skipped 1 previous similar message [12165.603813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12165.613601] Lustre: Skipped 1 previous similar message [12166.111473] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12166.333809] Lustre: server umount lustre-OST0000 complete [12170.722844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.731979] Lustre: Skipped 1 previous similar message [12175.844616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.853483] Lustre: Skipped 3 previous similar messages [12176.564131] Lustre: server umount lustre-MDT0000 complete [12180.986738] LustreError: 328147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12181.015546] LustreError: 328147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12181.335171] LustreError: 328132:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316404 with bad export cookie 9482219294816524580 [12181.345364] LustreError: 328132:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [12181.349798] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12181.743887] Lustre: server umount lustre-MDT0001 complete [12194.162633] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [12197.812966] Key type lgssc unregistered [12198.060231] LNet: 333097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12198.074607] LNetError: 333097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12198.110798] LNet: Removed LNI 192.168.201.125@tcp [12199.401351] Key type .llcrypt unregistered [12199.403618] Key type ._llcrypt unregistered [12245.115602] Key type ._llcrypt registered [12245.117967] Key type .llcrypt registered [12245.214935] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12246.449854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12246.478498] alg: No test for adler32 (adler32-zlib) [12247.878780] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [12248.123983] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [12249.847425] Key type lgssc registered [12251.055833] Lustre: Echo OBD driver; http://www.lustre.org/ [12259.955140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12260.003797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12261.485223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12265.617760] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12271.082567] LustreError: 335400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12271.098179] LustreError: 335400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12273.746134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.226726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12279.797471] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12282.504625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12288.961687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12289.351474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12294.654494] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12299.581089] LustreError: 336891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.115048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12302.285675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12306.431061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12306.458765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12308.126415] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12317.529740] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12325.187619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12340.706374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12340.714408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12340.737959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12342.245096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12342.248729] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12342.273354] Lustre: Skipped 2 previous similar messages [12344.070114] Lustre: server umount lustre-MDT0000 complete [12347.386404] LustreError: 335395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12347.404159] LustreError: 335395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12348.401087] LustreError: 335382:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316571 with bad export cookie 14385158695031719310 [12348.419495] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12348.422814] LustreError: 335382:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [12348.846465] Lustre: server umount lustre-MDT0001 complete [12363.091264] Lustre: server umount lustre-OST0000 complete [12377.693056] Lustre: server umount lustre-OST0001 complete [12398.078771] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12410.293791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.988401] LustreError: 340664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12411.142821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12416.483451] LustreError: 340665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12416.512024] LustreError: 340665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12416.971747] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12426.271137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.678789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12431.990674] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12435.133250] Lustre: 341804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12442.704710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12443.116599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12451.242355] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12451.334432] LustreError: 342159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12451.347256] LustreError: 342159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12457.503789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12463.629860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12469.260773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12471.132911] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12480.106582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12484.476464] Lustre: 343676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12489.722867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12489.747688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12489.750378] Lustre: Skipped 3 previous similar messages [12489.770398] Lustre: Skipped 6 previous similar messages [12494.818187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12494.838619] Lustre: Skipped 3 previous similar messages [12495.358298] Lustre: server umount lustre-MDT0000 complete [12499.262722] LustreError: 340647:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316722 with bad export cookie 14385158695031721011 [12499.268372] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12499.278350] LustreError: 340647:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [12499.776569] Lustre: server umount lustre-MDT0001 complete [12514.846524] Lustre: server umount lustre-OST0000 complete [12516.836636] Lustre: 334664:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316723/real 1786316723] req@ffff8a286428d500 x1873088580011008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786316739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12516.863452] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12520.132419] Lustre: server umount lustre-OST0001 complete [12524.587311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12526.585137] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12529.274483] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 19:05:50 (1786316750) [12555.593767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12566.219885] loop0: detected capacity change from 0 to 1099511627776 [12567.080429] SGI XFS with ACLs, security attributes, quota, no debug enabled [12567.087414] *** VALIDATE xfs *** [12567.095656] XFS (loop0): Mounting V5 Filesystem [12567.141098] XFS (loop0): Ending clean mount [12568.758611] loop1: detected capacity change from 0 to 18691697672192 [12691.998389] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [12705.257885] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [12748.788130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12749.065191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12749.092635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12749.158978] Lustre: lustre-MDT0000: new disk, initializing [12749.234240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12749.244839] Lustre: Skipped 1 previous similar message [12749.261497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12754.972479] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12767.660322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12767.826800] Lustre: 348815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12767.885837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12767.889149] Lustre: Skipped 1 previous similar message [12767.959635] Lustre: lustre-MDT0001: new disk, initializing [12768.110268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12768.125792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12773.789404] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12784.339757] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12791.191805] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12799.767712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12800.167849] Lustre: lustre-OST0000: new disk, initializing [12800.171927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12800.177658] Lustre: 350784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12801.954154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12801.968092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12802.271078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12808.806869] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12822.388934] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12829.387597] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12829.616599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12835.112873] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12835.532802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12838.367970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12838.377845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.403600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12842.987081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12848.102090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.105554] Lustre: Skipped 3 previous similar messages [12852.191366] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12852.590371] Lustre: server umount lustre-OST0000 complete [12858.337541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12858.345610] Lustre: Skipped 2 previous similar messages [12863.456141] LustreError: 348828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12863.462720] Lustre: server umount lustre-MDT0000 complete [12863.469097] LustreError: 348828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12867.416915] LustreError: 348807:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317090 with bad export cookie 14385158695031722565 [12867.425944] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.445369] LustreError: 348807:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [12868.019969] Lustre: server umount lustre-MDT0001 complete [12878.101849] XFS (loop0): Unmounting Filesystem [12883.784436] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 19:11:45 (1786317105) [12891.529305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12891.976297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12891.982659] Lustre: Skipped 2 previous similar messages [12897.278219] LustreError: 353349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.300137] LustreError: 353349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12897.411956] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12907.283041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12912.741852] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12920.967699] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12927.773218] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12935.010131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12942.279116] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [12950.979650] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12964.328131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12964.335065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12964.351358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12964.356894] Lustre: Skipped 2 previous similar messages [12968.427144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12969.881471] Lustre: server umount lustre-OST0000 complete [12980.620375] Lustre: server umount lustre-MDT0000 complete [12983.777659] LustreError: 353350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.818920] LustreError: 353350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12985.619437] LustreError: 355192:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317208 with bad export cookie 14385158695031723447 [12985.625903] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12985.637885] LustreError: 355192:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [12986.086239] Lustre: server umount lustre-MDT0001 complete [13000.368418] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [13003.553983] Key type lgssc unregistered [13003.994930] LNet: 356984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13004.003449] LNetError: 356984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13004.034802] LNet: Removed LNI 192.168.201.125@tcp [13005.255181] Key type .llcrypt unregistered [13005.257892] Key type ._llcrypt unregistered [13032.957656] Key type ._llcrypt registered [13032.959369] Key type .llcrypt registered [13033.080500] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [13034.195979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13034.229968] alg: No test for adler32 (adler32-zlib) [13035.755919] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [13035.963784] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [13037.631194] Key type lgssc registered [13038.776467] Lustre: Echo OBD driver; http://www.lustre.org/ [13046.987250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13047.042323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13048.474064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13053.174936] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13058.021651] LustreError: 358552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.034970] LustreError: 358552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13061.440628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13061.844278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13066.640357] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13075.134654] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13080.458177] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13088.618231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13088.973643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13096.607177] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13104.337187] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13111.787496] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13111.797675] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13112.464073] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [13112.476828] Lustre: Skipped 1 previous similar message [13113.879982] Lustre: server umount lustre-MDT0001 complete [13116.897670] LustreError: 358549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13116.921890] LustreError: 358549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13122.019066] LustreError: 358553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13122.029816] LustreError: 358553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13126.428836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13126.542435] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13126.546958] Lustre: Found index 1 for lustre-MDT0001, updating log [13126.598886] Lustre: 358541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13127.110572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13132.307186] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13132.353784] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13135.876569] LustreError: 362150:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13135.884341] LustreError: 362150:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13135.899482] LustreError: 358545:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13135.959537] LustreError: 362153:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -17 [13135.968658] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13135.968658] [13437.412990] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13437.419962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13437.432662] Lustre: Skipped 2 previous similar messages [13440.128602] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [13440.146928] Lustre: Skipped 1 previous similar message [13445.218276] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [13445.226242] Lustre: Skipped 2 previous similar messages [13451.231227] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13451.519794] Lustre: server umount lustre-MDT0001 complete [13452.773698] LustreError: 360997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13452.782839] LustreError: 360997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13457.892241] LustreError: 358553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13457.929155] LustreError: 358553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13464.043737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13464.294913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13464.296794] Lustre: Found index 1 for lustre-MDT0001, updating log [13464.348980] Lustre: 358540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13464.645822] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13469.678533] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13471.299579] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13473.258325] LustreError: 370111:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13473.261199] LustreError: 370111:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13473.264631] LustreError: 358545:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13473.267456] LustreError: 358545:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13473.327955] LustreError: 370119:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13473.337180] LustreError: 370119:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13473.343563] LustreError: 370119:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -17 [13473.354404] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.125@tcp [13473.354404] [13778.054906] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.25@tcp (stopping) [13778.060499] Lustre: Skipped 3 previous similar messages [13781.986709] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13781.995666] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13791.203050] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13791.412155] Lustre: server umount lustre-MDT0001 complete [13792.231843] LustreError: 358553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.253148] LustreError: 358553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13802.161312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13802.445989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13802.455836] Lustre: Found index 1 for lustre-MDT0001, updating log [13802.571771] Lustre: 358540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13803.021311] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13807.749940] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [13808.112029] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13811.696323] LustreError: 378250:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13811.702512] LustreError: 378250:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13811.708599] LustreError: 358545:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13811.716501] LustreError: 358545:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13811.778258] LustreError: 378253:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13811.792581] LustreError: 378253:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -114 [13811.802525] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13811.802525] [14130.667563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14130.672344] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14130.675605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14130.675613] Lustre: Skipped 6 previous similar messages [14130.723640] Lustre: Skipped 2 previous similar messages [14136.974731] Lustre: server umount lustre-MDT0000 complete [14140.905206] LustreError: 376034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14140.946705] LustreError: 376034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [14141.663865] LustreError: 369762:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318364 with bad export cookie 13008842479431336386 [14141.691586] LustreError: 369762:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [14141.695820] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14146.020957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14146.030156] Lustre: Skipped 1 previous similar message [14148.515591] Lustre: server umount lustre-MDT0001 complete [14152.946657] Lustre: server umount lustre-OST0000 complete [14163.710217] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [14175.156327] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [14215.856729] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:33:57 (1786318437) [14223.835390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14224.110884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14224.148407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14224.251107] Lustre: lustre-MDT0000: new disk, initializing [14224.342671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14224.428471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14230.132143] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14242.469601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14242.598765] Lustre: 388538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14242.656362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14242.664246] Lustre: Skipped 1 previous similar message [14242.750552] Lustre: lustre-MDT0001: new disk, initializing [14242.881241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14242.933222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14242.951199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14247.055516] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14259.882598] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14267.961535] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14276.786731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14277.146729] Lustre: lustre-OST0000: new disk, initializing [14277.150288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14277.158543] Lustre: 390509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14277.288218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14283.887412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14283.908811] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14284.057741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14284.486508] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [14294.381120] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14303.716552] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14303.721161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14303.747370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14303.757653] Lustre: Skipped 6 previous similar messages [14304.738441] LustreError: 389653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14304.747189] LustreError: 389653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14305.007177] Lustre: server umount lustre-MDT0000 complete [14309.583480] LustreError: 388531:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318532 with bad export cookie 13008842479431338339 [14309.590211] LustreError: 389345:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14309.603492] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14310.027809] Lustre: server umount lustre-MDT0001 complete [14315.918838] Lustre: server umount lustre-OST0000 complete [14349.907116] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [14360.280154] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [14399.208084] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14061 sec ========= 19:36:59 (1786318619) [14401.727516] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:37:02 (1786318622) === [14405.222811] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:37:06 (1786318626) === [14452.708274] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [14455.928453] Key type lgssc unregistered [14456.399720] LNet: 396879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14456.403335] LNetError: 396879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14456.412907] LNet: Removed LNI 192.168.201.125@tcp [14457.355232] Key type .llcrypt unregistered [14457.357737] Key type ._llcrypt unregistered