[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512288208 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007911] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010118] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013051] *** VALIDATE selinux *** [ 0.020215] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024622] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027061] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.029403] *** VALIDATE proc *** [ 0.031074] *** VALIDATE cgroup *** [ 0.031912] *** VALIDATE cgroup2 *** [ 0.033100] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041156] debug: unmapping init [mem 0xffffffff8a859000-0xffffffff8a860fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044549] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.046952] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.049970] ... event mask: 000000070000000f [ 0.050244] rcu: Hierarchical SRCU implementation. [ 0.052370] smp: Bringing up secondary CPUs ... [ 0.053487] x86: Booting SMP configuration: [ 0.054018] .... node #0, CPUs: #1 #2 #3 [ 0.057152] smp: Brought up 1 node, 4 CPUs [ 0.058937] smpboot: Max logical packages: 1 [ 0.059010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142234] node 0 deferred pages initialised in 80ms [ 0.146093] devtmpfs: initialized [ 0.147237] x86/mm: Memory block size: 128MB [ 0.149565] gcov: version magic: 0x41383552 [ 0.153242] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162324] pinctrl core: initialized pinctrl subsystem [ 0.165190] [ 0.166008] ************************************************************* [ 0.169014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172014] ** ** [ 0.174011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176013] ** ** [ 0.178014] ** This means that this kernel is built to expose internal ** [ 0.181013] ** IOMMU data structures, which may compromise security on ** [ 0.183013] ** your system. ** [ 0.186017] ** ** [ 0.188010] ** If you see this message and you are not debugging the ** [ 0.191011] ** kernel, report this immediately to your vendor! ** [ 0.194012] ** ** [ 0.196011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199012] ************************************************************* [ 0.202563] NET: Registered protocol family 16 [ 0.204404] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212509] cpuidle: using governor menu [ 0.213630] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216442] PCI: Using configuration type 1 for base access [ 0.218117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228138] cryptd: max_cpu_qlen set to 1000 [ 0.231224] ACPI: Added _OSI(Module Device) [ 0.233011] ACPI: Added _OSI(Processor Device) [ 0.235014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242582] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248390] ACPI: Interpreter enabled [ 0.250062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251013] ACPI: Using IOAPIC for interrupt routing [ 0.254102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257361] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266607] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281305] acpiphp: Slot [2] registered [ 0.282110] acpiphp: Slot [5] registered [ 0.284104] acpiphp: Slot [6] registered [ 0.285103] acpiphp: Slot [7] registered [ 0.287144] acpiphp: Slot [8] registered [ 0.288103] acpiphp: Slot [9] registered [ 0.289077] acpiphp: Slot [10] registered [ 0.289990] acpiphp: Slot [3] registered [ 0.291070] acpiphp: Slot [4] registered [ 0.293075] acpiphp: Slot [11] registered [ 0.294091] acpiphp: Slot [12] registered [ 0.295086] acpiphp: Slot [13] registered [ 0.296076] acpiphp: Slot [14] registered [ 0.297075] acpiphp: Slot [15] registered [ 0.299142] acpiphp: Slot [16] registered [ 0.300058] acpiphp: Slot [17] registered [ 0.301074] acpiphp: Slot [18] registered [ 0.302071] acpiphp: Slot [19] registered [ 0.303096] acpiphp: Slot [20] registered [ 0.304061] acpiphp: Slot [21] registered [ 0.306076] acpiphp: Slot [22] registered [ 0.307083] acpiphp: Slot [23] registered [ 0.308082] acpiphp: Slot [24] registered [ 0.310096] acpiphp: Slot [25] registered [ 0.311094] acpiphp: Slot [26] registered [ 0.313114] acpiphp: Slot [27] registered [ 0.315095] acpiphp: Slot [28] registered [ 0.316098] acpiphp: Slot [29] registered [ 0.318107] acpiphp: Slot [30] registered [ 0.320103] acpiphp: Slot [31] registered [ 0.321061] PCI host bridge to bus 0000:00 [ 0.323021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345915] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350333] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377543] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380945] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383092] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387010] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409955] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416065] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.451188] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510217] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.535015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.558015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.571640] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.586014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.601015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.609105] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.614026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653245] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.664017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.695980] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.699389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.704368] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.710363] iommu: Default domain type: Passthrough [ 0.712949] SCSI subsystem initialized [ 0.714148] ACPI: bus type USB registered [ 0.715095] usbcore: registered new interface driver usbfs [ 0.717070] usbcore: registered new interface driver hub [ 0.719116] usbcore: registered new device driver usb [ 0.721156] pps_core: LinuxPPS API ver. 1 registered [ 0.723011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.726057] PTP clock support registered [ 0.728086] EDAC MC: Ver: 3.0.0 [ 0.730063] PCI: Using ACPI for IRQ routing [ 0.732826] NetLabel: Initializing [ 0.734011] NetLabel: domain hash size = 128 [ 0.735018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737088] NetLabel: unlabeled traffic allowed by default [ 0.740090] vgaarb: loaded [ 0.741172] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.743012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750004] clocksource: Switched to clocksource kvm-clock [ 0.858615] VFS: Disk quotas dquot_6.6.0 [ 0.860295] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863100] *** VALIDATE ramfs *** [ 0.864554] *** VALIDATE hugetlbfs *** [ 0.866235] pnp: PnP ACPI init [ 0.868886] pnp: PnP ACPI: found 6 devices [ 0.895658] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.899180] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.901408] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.903828] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906297] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909013] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.911869] NET: Registered protocol family 2 [ 0.914233] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.918930] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.922666] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.927986] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931500] TCP: Hash tables configured (established 65536 bind 65536) [ 0.934272] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.937237] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.940437] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.943336] NET: Registered protocol family 1 [ 0.946236] RPC: Registered named UNIX socket transport module. [ 0.948231] RPC: Registered udp transport module. [ 0.949874] RPC: Registered tcp transport module. [ 0.951375] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.953586] NET: Registered protocol family 44 [ 0.955300] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.957566] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.959727] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962119] PCI: CLS 0 bytes, default 64 [ 0.963742] Unpacking initramfs... [ 2.344635] debug: unmapping init [mem 0xffff964cbcc54000-0xffff964cbffbffff] [ 2.348493] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.350962] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.354092] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.860992] Initialise system trusted keyrings [ 2.862716] Key type blacklist registered [ 2.865587] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.874667] zbud: loaded [ 2.877763] *** VALIDATE nfs *** [ 2.879012] *** VALIDATE nfs4 *** [ 2.880691] pstore: using deflate compression [ 2.884927] Platform Keyring initialized [ 2.985735] NET: Registered protocol family 38 [ 2.987502] Key type asymmetric registered [ 2.988853] Asymmetric key parser 'x509' registered [ 2.990824] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.993126] io scheduler mq-deadline registered [ 2.994610] io scheduler kyber registered [ 2.996100] io scheduler bfq registered [ 2.997697] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.000459] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.003186] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.005803] ACPI: Power Button [PWRF] [ 3.011303] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.017935] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.045229] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.059149] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.096595] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.128675] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.162304] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.167565] Non-volatile memory driver v1.3 [ 3.169534] Linux agpgart interface v0.103 [ 3.200890] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.204034] vda: detected capacity change from 0 to 74702848 [ 3.222618] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.225542] vdb: detected capacity change from 0 to 1073741824 [ 3.241684] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.244485] vdc: detected capacity change from 0 to 2621440000 [ 3.259359] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.261576] vdd: detected capacity change from 0 to 2621440000 [ 3.273454] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.275885] vde: detected capacity change from 0 to 4294967296 [ 3.290198] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.293686] vdf: detected capacity change from 0 to 4294967296 [ 3.310942] libphy: Fixed MDIO Bus: probed [ 3.316399] usbcore: registered new interface driver usbserial_generic [ 3.319322] usbserial: USB Serial support registered for generic [ 3.322052] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.326711] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.328744] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.331587] mousedev: PS/2 mouse device common for all mice [ 3.334162] rtc_cmos 00:05: RTC can wake from S4 [ 3.337104] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.340250] rtc_cmos 00:05: registered as rtc0 [ 3.342433] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.343761] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.346856] intel_pstate: CPU model not supported [ 3.351243] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.354898] hid: raw HID events driver (C) Jiri Kosina [ 3.356845] usbcore: registered new interface driver usbhid [ 3.358537] usbhid: USB HID core driver [ 3.359916] drop_monitor: Initializing network drop monitor service [ 3.361809] Initializing XFRM netlink socket [ 3.363543] NET: Registered protocol family 10 [ 3.366238] Segment Routing with IPv6 [ 3.367619] NET: Registered protocol family 17 [ 3.370302] mpls_gso: MPLS GSO support [ 3.376630] RAS: Correctable Errors collector initialized. [ 3.378690] AVX version of gcm_enc/dec engaged. [ 3.380393] AES CTR mode by8 optimization enabled [ 3.465564] sched_clock: Marking stable (3465536864, 0)->(4305037793, -839500929) [ 3.469376] registered taskstats version 1 [ 3.471312] Loading compiled-in X.509 certificates [ 3.473858] zswap: loaded using pool lzo/zbud [ 3.503932] Key type big_key registered [ 3.520064] Key type encrypted registered [ 3.521826] ima: No TPM chip found, activating TPM-bypass! [ 3.524323] ima: Allocated hash algorithm: sha1 [ 3.526117] ima: No architecture policies found [ 3.528180] evm: Initialising EVM extended attributes: [ 3.530323] evm: security.selinux [ 3.531656] evm: security.ima [ 3.532405] evm: security.capability [ 3.534169] evm: HMAC attrs: 0x1 [ 3.536684] rtc_cmos 00:05: setting system clock to 2026-08-17 13:53:00 UTC (1786974780) [ 3.543436] debug: unmapping init [mem 0xffffffff8b803000-0xffffffff8b9fffff] [ 3.546784] debug: unmapping init [mem 0xffffffff8a582000-0xffffffff8a858fff] [ 3.560096] Write protecting the kernel read-only data: 28672k [ 3.563879] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 3.567150] debug: unmapping init [mem 0xffffffff89514000-0xffffffff895fffff] [ 3.606197] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.616827] systemd[1]: Detected virtualization kvm. [ 3.619059] systemd[1]: Detected architecture x86-64. [ 3.621451] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.648688] systemd[1]: No hostname configured. [ 3.650592] systemd[1]: Set hostname to . [ 3.652888] random: systemd: uninitialized urandom read (16 bytes read) [ 3.655393] systemd[1]: Initializing machine ID from random generator. [ 3.691296] random: ln: uninitialized urandom read (6 bytes read) [ 3.792168] random: systemd: uninitialized urandom read (16 bytes read) [ 3.795635] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.805420] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.809927] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.453907] device-mapper: uevent: version 1.0.3 [ 4.456767] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.321418] random: fast init done [ 5.347843] virtio_net virtio0 ens2: renamed from eth0 [ 5.388762] scsi host0: ata_piix [ 5.435151] scsi host1: ata_piix [ 5.447516] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.450242] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.231399] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.040099] random: crng init done [ 10.041535] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.570888] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.835324] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.178476] SELinux: Disabled at runtime. [ 12.248847] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.258164] systemd[1]: Detected virtualization kvm. [ 12.260125] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.838428] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.841337] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.845383] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.851332] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.854099] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.860236] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.868429] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ 12.898464] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.313907] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.774313] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.806786] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.939930] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.962548] EDAC sbridge: Ver: 1.1.2 [ 15.726591] Key type dns_resolver registered [ 16.065916] NFS: Registering the id_resolver key type [ 16.068470] Key type id_resolver registered [ 16.070181] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg336-server login: [ 42.928308] libcfs: loading out-of-tree module taints kernel. [ 42.948358] Key type ._llcrypt registered [ 42.949929] Key type .llcrypt registered [ 43.001932] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 51.663496] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 52.479764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.491626] alg: No test for adler32 (adler32-zlib) [ 54.424046] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 55.845625] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 57.784055] Key type lgssc registered [ 60.220789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.476645] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 128.709443] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 145.051458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.101326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.590540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.624241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.693209] Lustre: lustre-MDT0000: new disk, initializing [ 146.828257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.861526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.509921] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.420371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.543310] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 166.575903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 166.581384] Lustre: Skipped 1 previous similar message [ 166.715330] Lustre: lustre-MDT0001: new disk, initializing [ 166.814478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.845293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.860150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 171.679458] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.838840] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 182.784334] hrtimer: interrupt took 3127661 ns [ 186.036240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.292338] Lustre: lustre-OST0000: new disk, initializing [ 186.296631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.311133] Lustre: 8443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.410867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.823515] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.128261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.150909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.296851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.628489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.754151] Lustre: lustre-OST0001: new disk, initializing [ 209.757306] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.769450] Lustre: 9515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.827121] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.115754] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.134428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.156743] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.217850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.212681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.246214] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.659790] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing check_logdir /tmp/testlogs/ [ 247.492946] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing yml_node [ 251.994490] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 254.592229] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 257.303979] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 259.088796] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 09:57:15 EDT 2026 [ 277.204921] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 278.598970] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 294.886962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.896976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.906837] Lustre: Skipped 2 previous similar messages [ 294.907352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 294.915431] Lustre: Skipped 3 previous similar messages [ 300.000742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.003673] Lustre: Skipped 3 previous similar messages [ 300.624640] Lustre: server umount lustre-MDT0000 complete [ 304.491111] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975081 with bad export cookie 5130219541405398509 [ 304.503214] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.518729] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 305.121434] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.129354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.150384] Lustre: Skipped 2 previous similar messages [ 305.171415] Lustre: Skipped 1 previous similar message [ 309.152273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.162234] Lustre: Skipped 1 previous similar message [ 311.082562] Lustre: server umount lustre-MDT0001 complete [ 315.411086] Lustre: server umount lustre-OST0000 complete [ 319.679138] Lustre: server umount lustre-OST0001 complete [ 327.445618] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 340.816045] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 381.221735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 381.613316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 381.658785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 381.881523] Lustre: lustre-MDT0000: new disk, initializing [ 382.006612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.027502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 386.284155] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 397.819784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.084879] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 398.161365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 398.168410] Lustre: Skipped 1 previous similar message [ 398.360752] Lustre: lustre-MDT0001: new disk, initializing [ 398.510713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.562178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.579574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 403.467430] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 414.087296] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 419.358129] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 425.595537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.955954] Lustre: lustre-OST0000: new disk, initializing [ 425.964252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.972280] Lustre: 17039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 426.129054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.187687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 427.211496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.418773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.397422] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 442.037972] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.482390] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.780950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.662817] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.141117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 458.209667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 458.214199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.231390] LustreError: Skipped 1 previous similar message [ 458.258524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.822594] Lustre: server umount lustre-OST0000 complete [ 468.453040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.456096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 468.474336] Lustre: Skipped 2 previous similar messages [ 468.489085] Lustre: Skipped 2 previous similar messages [ 473.377125] Lustre: server umount lustre-MDT0000 complete [ 473.572623] LustreError: 15914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.592555] LustreError: 15914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 477.122306] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975254 with bad export cookie 5130219541405400042 [ 477.126115] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.130277] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 477.459334] Lustre: server umount lustre-MDT0001 complete [ 482.827834] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:00:59 (1786975259) [ 490.061154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.640058] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.758137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.396953] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 496.099705] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.198271] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.650252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.324406] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 516.647936] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 522.724594] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 530.039760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 537.421728] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 546.721095] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 553.903599] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 561.123498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.135423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.175228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.262411] Lustre: server umount lustre-MDT0000 complete [ 570.348407] LustreError: 21614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.382660] LustreError: 21614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 571.546933] LustreError: 19298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975348 with bad export cookie 5130219541405400924 [ 571.554495] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.566903] LustreError: 19298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 571.975250] Lustre: server umount lustre-MDT0001 complete [ 586.905865] Lustre: server umount lustre-OST0000 complete [ 611.178570] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 622.360193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.942560] LustreError: 23700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.030375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.042456] Lustre: Skipped 2 previous similar messages [ 628.393476] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 638.206208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.273338] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 647.994215] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 657.384200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.138846] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 674.957839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 675.221824] Lustre: lustre-OST0001: new disk, initializing [ 675.239683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 675.254091] Lustre: 26016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 676.471458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 676.492126] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 676.685961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 682.552415] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 693.368506] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 697.476338] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 712.688073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.693031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.715419] Lustre: Skipped 5 previous similar messages [ 712.733442] Lustre: Skipped 6 previous similar messages [ 717.673911] Lustre: server umount lustre-MDT0000 complete [ 717.803595] LustreError: 23695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.838762] LustreError: 23695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 721.704684] LustreError: 23680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975498 with bad export cookie 5130219541405401771 [ 721.720222] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.721423] LustreError: 23680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.969549] Lustre: server umount lustre-MDT0001 complete [ 735.278926] Lustre: server umount lustre-OST0000 complete [ 738.275070] Lustre: 3638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786975499/real 1786975499] req@ffff964c054eca80 x1873778921684992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786975515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 738.317389] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.796595] Lustre: server umount lustre-OST0001 complete [ 758.453233] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 768.390368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.795081] LustreError: 29036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.907302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.913100] Lustre: Skipped 3 previous similar messages [ 774.517199] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 783.395866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.801858] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 792.338718] Lustre: 30184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 800.928487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.049408] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 818.273920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 825.865899] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 834.562188] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 838.767176] Lustre: 32065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 849.388251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.392239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.412767] Lustre: Skipped 2 previous similar messages [ 849.426306] Lustre: Skipped 2 previous similar messages [ 852.073909] Lustre: server umount lustre-MDT0000 complete [ 854.509881] LustreError: 29033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.526839] LustreError: 29033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 855.859499] LustreError: 29019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975632 with bad export cookie 5130219541405403066 [ 855.862703] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.876719] LustreError: 29019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 856.232893] Lustre: server umount lustre-MDT0001 complete [ 870.501585] Lustre: server umount lustre-OST0000 complete [ 885.270822] Lustre: server umount lustre-OST0001 complete [ 897.880493] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:07:53 (1786975673) [ 927.605447] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 939.070880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.413966] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 955.339672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.754829] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 965.012129] Lustre: 36022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 972.273413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.562525] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 984.120495] LustreError: 36817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.137305] LustreError: 36817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 989.784050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.172809] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1005.085152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1009.417683] Lustre: 37903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1015.783474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.794363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.811441] Lustre: Skipped 4 previous similar messages [ 1015.819526] Lustre: Skipped 2 previous similar messages [ 1021.751967] Lustre: server umount lustre-MDT0000 complete [ 1025.757986] LustreError: 34852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975802 with bad export cookie 5130219541405404172 [ 1025.760326] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.769646] LustreError: 34852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1026.113445] Lustre: server umount lustre-MDT0001 complete [ 1036.414355] Lustre: server umount lustre-OST0000 complete [ 1040.660594] Lustre: server umount lustre-OST0001 complete [ 1049.648816] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:10:26 (1786975826) [ 1067.265879] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1077.941050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.401385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.426420] Lustre: Skipped 7 previous similar messages [ 1083.227443] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1092.511411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.199474] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1101.328177] Lustre: 41194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1108.505900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1116.655761] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1128.009211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.468846] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1142.595519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1146.554808] Lustre: 43071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1147.483581] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1147.488971] Lustre: Skipped 1 previous similar message [ 1154.535970] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.536440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1154.552164] Lustre: Skipped 4 previous similar messages [ 1156.974976] Lustre: server umount lustre-MDT0000 complete [ 1160.304310] LustreError: 41577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786975937 with bad export cookie 5130219541405405278 [ 1160.315442] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.320722] LustreError: 41577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1166.937253] Lustre: server umount lustre-MDT0001 complete [ 1176.399330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.918531] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1195.058714] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1201.061604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.602817] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1206.785023] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1206.795490] Lustre: Skipped 1 previous similar message [ 1214.534958] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.169165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1221.131896] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1222.534681] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1227.242769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1232.975234] Lustre: server umount lustre-MDT0000 complete [ 1236.154958] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976013 with bad export cookie 5130219541405406377 [ 1236.166234] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.169782] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1236.588236] Lustre: server umount lustre-MDT0001 complete [ 1250.737214] Lustre: server umount lustre-OST0000 complete [ 1253.856190] Lustre: 3637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786976014/real 1786976014] req@ffff964d04552d80 x1873778921877760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786976030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1255.074354] Lustre: server umount lustre-OST0001 complete [ 1263.929116] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:14:00 (1786976040) [ 1279.551574] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1291.841919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.765884] LustreError: 47678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.788792] LustreError: 47678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages [ 1297.347569] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1308.590781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.557366] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1318.333358] Lustre: 48835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1324.761312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1331.925356] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1341.677422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.319754] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1356.200985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1360.855295] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1372.645126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.658472] Lustre: Skipped 26 previous similar messages [ 1373.972221] Lustre: server umount lustre-MDT0000 complete [ 1377.148156] LustreError: 47659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976154 with bad export cookie 5130219541405407203 [ 1377.154885] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.159294] LustreError: 47659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1377.450717] Lustre: server umount lustre-MDT0001 complete [ 1391.735481] Lustre: server umount lustre-OST0000 complete [ 1393.636126] Lustre: 3635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786976154/real 1786976154] req@ffff964d3d3cbb80 x1873778921932032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786976170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1395.559573] Lustre: server umount lustre-OST0001 complete [ 1404.852790] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:16:21 (1786976181) [ 1407.353643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1408.753788] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:16:25 (1786976185) [ 1417.863668] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 1426.561835] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1447.168213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.403709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1447.459548] Lustre: lustre-MDT0000: new disk, initializing [ 1447.524448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1451.869121] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1461.678194] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.884093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.208181] Lustre: lustre-OST0000: new disk, initializing [ 1469.211568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1469.215841] Lustre: Skipped 1 previous similar message [ 1469.227698] Lustre: 54907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1470.516594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1470.539594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1470.587210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1476.148595] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1487.660739] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1491.233678] Lustre: DEBUG MARKER: create test files [ 1531.874956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1531.888511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.900242] Lustre: Skipped 12 previous similar messages [ 1535.345526] Lustre: server umount lustre-OST0000 complete [ 1539.352224] Lustre: server umount lustre-MDT0000 complete [ 1551.120742] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1554.237287] Key type lgssc unregistered [ 1554.556174] LNet: 56546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.569295] LNetError: 56546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.591911] LNet: Removed LNI 192.168.203.136@tcp [ 1555.491130] Key type .llcrypt unregistered [ 1555.493109] Key type ._llcrypt unregistered [ 1585.314487] Key type ._llcrypt registered [ 1585.316530] Key type .llcrypt registered [ 1585.445846] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1587.166467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.257963] alg: No test for adler32 (adler32-zlib) [ 1588.436711] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1588.780609] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1590.575897] Key type lgssc registered [ 1592.679736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.063945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1605.101091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.937351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.953824] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1619.501719] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1626.208522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.645878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.786051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1631.933565] Lustre: 59280:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1631.940429] Lustre: 59280:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1631.947495] Lustre: 59280:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1631.955632] Lustre: 59280:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1631.971601] Lustre: 59280:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1631.983471] Lustre: 59280:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1632.892485] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1639.403339] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1645.573508] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1651.089106] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1657.312448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1657.318441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.328161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.511927] Lustre: server umount lustre-OST0000 complete [ 1663.158867] Lustre: server umount lustre-MDT0000 complete [ 1673.927955] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1676.832510] Key type lgssc unregistered [ 1677.222927] LNet: 60892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.229612] LNetError: 60892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.260265] LNet: Removed LNI 192.168.203.136@tcp [ 1678.018901] Key type .llcrypt unregistered [ 1678.022095] Key type ._llcrypt unregistered [ 1706.916589] Key type ._llcrypt registered [ 1706.919345] Key type .llcrypt registered [ 1707.033775] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1707.835800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.069897] alg: No test for adler32 (adler32-zlib) [ 1709.134926] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1709.392450] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1711.047705] Key type lgssc registered [ 1712.172414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.908882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.938743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.281944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.375417] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1731.689160] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1737.649684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.926279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.828782] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1746.132439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1746.243924] Lustre: 63705:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1746.262850] Lustre: 63705:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1746.270085] Lustre: 63705:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1746.282471] Lustre: 63705:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1746.298759] Lustre: 63705:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1746.314478] Lustre: 63705:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1751.976384] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1758.669852] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1772.009703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.021804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1772.056374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.855451] Lustre: server umount lustre-OST0000 complete [ 1781.288602] Lustre: server umount lustre-MDT0000 complete [ 1792.765424] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 1795.977396] Key type lgssc unregistered [ 1796.335850] LNet: 65270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.341751] LNetError: 65270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.364441] LNet: Removed LNI 192.168.203.136@tcp [ 1797.310469] Key type .llcrypt unregistered [ 1797.312828] Key type ._llcrypt unregistered [ 1815.943835] Key type ._llcrypt registered [ 1815.947748] Key type .llcrypt registered [ 1816.059518] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 1832.011061] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 1833.081739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.103101] alg: No test for adler32 (adler32-zlib) [ 1834.139756] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1834.379392] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 1836.063199] Key type lgssc registered [ 1837.141472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1871.580535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.624378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.867997] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1872.911631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1873.004727] Lustre: lustre-MDT0000: new disk, initializing [ 1873.087372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.105649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1877.934883] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1888.639250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.757928] Lustre: 68424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1888.840884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1888.850172] Lustre: Skipped 1 previous similar message [ 1888.944678] Lustre: lustre-MDT0001: new disk, initializing [ 1889.015962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.060175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1889.080929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1893.250766] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1903.210859] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.356095] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1916.197162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.435947] Lustre: lustre-OST0000: new disk, initializing [ 1916.444130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1916.452194] Lustre: 70392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1916.535635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.173681] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 1923.633844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1923.644652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1923.789906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1932.059324] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1936.543729] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1936.764824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1940.919741] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1941.111820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1944.549219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.550509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1944.552546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.560241] Lustre: Skipped 1 previous similar message [ 1949.687603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.691366] Lustre: Skipped 2 previous similar messages [ 1954.788019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.797406] Lustre: Skipped 1 previous similar message [ 1957.343157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1957.527859] Lustre: server umount lustre-OST0000 complete [ 1965.024611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.048590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.012971] Lustre: server umount lustre-MDT0000 complete [ 1971.606100] LustreError: 68416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976748 with bad export cookie 5439995506373075290 [ 1971.610668] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1971.618966] LustreError: 68416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1971.941501] Lustre: server umount lustre-MDT0001 complete [ 1980.960486] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:25:57 (1786976757) [ 1992.337144] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2035.536492] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 2047.655825] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2084.681536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.937512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2084.980797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2085.097865] Lustre: lustre-MDT0000: new disk, initializing [ 2085.175126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.203866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2089.998560] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2100.793221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.918912] Lustre: 76413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2100.977534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2100.985889] Lustre: Skipped 1 previous similar message [ 2101.019118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2101.025101] Lustre: Skipped 1 previous similar message [ 2101.084960] Lustre: lustre-MDT0001: new disk, initializing [ 2101.156377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2101.189843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2101.205932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2106.174643] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2116.373596] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2122.831382] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2129.959218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.321714] Lustre: lustre-OST0000: new disk, initializing [ 2130.329753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2130.341301] Lustre: 78383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2130.428661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.734701] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2138.165136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2138.179884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2138.310473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2146.692234] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2151.787572] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2152.013343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2158.666758] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2158.886562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2164.202352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.208604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.212048] LustreError: Skipped 2 previous similar messages [ 2164.251317] Lustre: Skipped 1 previous similar message [ 2164.263472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2167.404274] Lustre: server umount lustre-OST0000 complete [ 2172.896112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.904247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.921017] Lustre: Skipped 1 previous similar message [ 2177.966303] Lustre: server umount lustre-MDT0000 complete [ 2179.556546] LustreError: 76424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.594273] LustreError: 76424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2182.345380] LustreError: 76406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786976959 with bad export cookie 5439995506373076172 [ 2182.357197] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.361841] LustreError: 76406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2182.622213] Lustre: server umount lustre-MDT0001 complete [ 2191.873327] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:29:28 (1786976968) [ 2198.321716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.837282] LustreError: 80798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.944446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.168547] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2204.135516] LustreError: 80799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.227883] LustreError: 80798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.849850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.352555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2213.367158] Lustre: Skipped 4 previous similar messages [ 2219.142723] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2229.079829] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2234.932652] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2242.823543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2243.294471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.302901] Lustre: Skipped 1 previous similar message [ 2249.705197] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2257.206157] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2261.010159] Lustre: 83466:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting cef5a3d5-7f78-42f5-8b8d-ef61bb2261e6 at adminstrative request [ 2264.040858] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2264.061933] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2274.281066] Lustre: MGS: Client cef5a3d5-7f78-42f5-8b8d-ef61bb2261e6 (at 0@lo) reconnecting [ 2274.301274] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x4b7ebf173916a837 to 0x4b7ebf173916aade [ 2274.312884] Lustre: MGC192.168.203.136@tcp: Connection restored to 0@lo (at 0@lo) [ 2298.954409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.148454] Lustre: lustre-OST0001: new disk, initializing [ 2299.153314] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2299.164749] Lustre: 83859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2306.062839] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2306.151287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2306.175211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2306.371472] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2317.693918] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2321.889261] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2321.891923] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.900426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2321.926620] Lustre: Skipped 2 previous similar messages [ 2327.297918] Lustre: server umount lustre-OST0001 complete [ 2331.617981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2331.632446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.154868] LustreError: 84530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.173380] LustreError: 84530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2337.573388] Lustre: server umount lustre-OST0000 complete [ 2341.856504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2347.780789] Lustre: server umount lustre-MDT0000 complete [ 2351.940449] LustreError: 80778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977128 with bad export cookie 5439995506373077726 [ 2351.941733] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2351.948758] LustreError: 80778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2352.341580] Lustre: server umount lustre-MDT0001 complete [ 2362.025523] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:32:18 (1786977138) [ 2363.659206] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2365.896657] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:32:22 (1786977142) [ 2367.522511] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2369.319376] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:32:25 (1786977145) [ 2370.681189] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2372.571323] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 10:32:29 (1786977149) [ 2374.120195] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2375.795389] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:32:32 (1786977152) [ 2377.446382] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2384.122603] loop: module loaded [ 2384.134696] loop0: detected capacity change from 0 to 4294967296 [ 2384.204674] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2386.067815] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2390.486631] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2394.454458] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2423.759363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.812608] LustreError: 87984:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2423.831819] LustreError: 87984:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2423.844051] LustreError: 87984:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2423.851844] LustreError: 87984:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2443.432113] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 2453.810215] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 2492.463100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.662390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2492.693229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2492.819710] Lustre: lustre-MDT0000: new disk, initializing [ 2492.869727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.875724] Lustre: Skipped 1 previous similar message [ 2492.885899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2498.045236] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2510.213135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.407839] Lustre: 90863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2510.603338] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2510.605570] Lustre: Skipped 13 previous similar messages [ 2510.764547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2516.881415] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2527.247483] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2533.153682] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2540.003790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2540.399965] Lustre: 92830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2547.178704] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2550.359104] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2550.548762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2558.988482] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2565.965719] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2566.256149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2572.603373] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2572.953985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2576.359983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.370092] Lustre: Skipped 3 previous similar messages [ 2576.370218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.678255] Lustre: server umount lustre-OST0000 complete [ 2592.560422] Lustre: server umount lustre-MDT0000 complete [ 2596.840740] LustreError: 90869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.864300] LustreError: 90869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2596.965771] LustreError: 90855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977373 with bad export cookie 5439995506373078335 [ 2596.975376] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.981284] LustreError: 90855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2597.312168] Lustre: server umount lustre-MDT0001 complete [ 2608.115995] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:36:23 (1786977383) [ 2616.645768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.214772] LustreError: 95243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.126983] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2635.370503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.819264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2635.831657] Lustre: Skipped 3 previous similar messages [ 2642.205570] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2651.595231] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2657.894374] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2665.187678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.160125] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2680.393116] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2685.708359] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2689.097945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2693.850102] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2694.084293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2701.497703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.771511] Lustre: lustre-OST0001: new disk, initializing [ 2701.775486] Lustre: Skipped 2 previous similar messages [ 2701.782345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2701.788339] Lustre: Skipped 3 previous similar messages [ 2701.798815] Lustre: 98636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2708.643553] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2709.611972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2709.624748] Lustre: Skipped 2 previous similar messages [ 2709.631714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2709.774230] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2718.496456] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2724.186201] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2724.439928] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2729.282320] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2729.617228] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2750.531397] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2777.987534] Lustre: Failing over lustre-MDT0000 [ 2778.468815] Lustre: server umount lustre-MDT0000 complete [ 2779.104982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2779.112666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.125629] Lustre: Skipped 3 previous similar messages [ 2779.149793] LustreError: 96499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.173883] LustreError: 96499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2798.052265] Lustre: 66650:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786977558/real 1786977558] req@0000000066722022 x1873780787135616/t0(0) o400->MGC192.168.203.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786977574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2798.089522] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.600779] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2804.605803] LDISKFS-fs (dm-0): recovery complete [ 2804.619052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.789255] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2814.713184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2814.720189] Lustre: 96358:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2814.989803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2815.073767] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2815.416670] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2815.443886] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2815.463706] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 2815.795753] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2815.815533] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2816.143262] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2816.169421] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2816.322869] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2816.532560] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2816.550362] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 2 previous similar messages [ 2816.879952] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2816.894396] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2817.227685] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2817.235575] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2818.549829] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2818.563489] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 5 previous similar messages [ 2818.910748] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2818.928344] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 2819.255141] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2819.262347] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 2822.755696] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2822.776650] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 2823.121825] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2823.149663] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2823.479175] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2823.486100] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2830.810687] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2830.828051] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 2831.152318] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2831.165023] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2831.496255] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2831.502349] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2847.016761] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2847.034339] Lustre: 101487:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 50 previous similar messages [ 2847.364905] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2847.371497] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 2847.695357] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2847.707511] LustreError: 101487:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 2874.135932] Lustre: 101487:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2874.151373] Lustre: 101487:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2874.168389] Lustre: 101487:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f066459e-b895-4e33-972f-cfe63c49b6f8@192.168.203.36@tcp [ 2874.182095] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2874.186267] Lustre: 101487:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000c77e3670 x1873780771200128/t0(8589934781) o101->f066459e-b895-4e33-972f-cfe63c49b6f8@192.168.203.36@tcp:626/0 lens 592/0 e 0 to 0 dl 1786977661 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2874.235395] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2874.246568] Lustre: lustre-MDT0000: Denying connection for new client f066459e-b895-4e33-972f-cfe63c49b6f8 (at 192.168.203.36@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2874.276824] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2874.325598] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2874.339558] Lustre: Skipped 2 previous similar messages [ 2874.342388] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2874.398673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2874.400886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2891.767335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.771569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.779807] Lustre: Skipped 4 previous similar messages [ 2891.796045] Lustre: Skipped 8 previous similar messages [ 2897.352571] Lustre: server umount lustre-OST0000 complete [ 2902.006617] LustreError: 97066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.047566] LustreError: 97066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 2907.489206] Lustre: server umount lustre-OST0001 complete [ 2915.298429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.000561] Lustre: server umount lustre-MDT0000 complete [ 2921.925082] LustreError: 95226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977698 with bad export cookie 5439995506373129547 [ 2921.929806] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2921.935553] LustreError: 95226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2922.337871] Lustre: server umount lustre-MDT0001 complete [ 2930.443034] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:41:47 (1786977707) [ 2938.007493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2938.014830] Lustre: *** cfs_fail_loc=197, val=0*** [ 2953.759568] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2958.880172] LustreError: 103460:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 2964.044548] Lustre: *** cfs_fail_loc=197, val=0*** [ 2964.051305] Lustre: Skipped 65 previous similar messages [ 2984.674737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.678340] Lustre: Skipped 3 previous similar messages [ 2990.282686] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 2997.248592] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.060889] Lustre: server umount lustre-OST0000 complete [ 3006.678594] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:43:03 (1786977783) [ 3026.522722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.347801] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3032.545530] LustreError: 105080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.564796] LustreError: 105080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3039.764809] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3043.827486] LustreError: 105110:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3043.842815] LustreError: 105110:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3043.849884] LustreError: 105110:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 3044.243789] Lustre: server umount lustre-MDT0000 complete [ 3077.150864] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 3086.520122] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3117.262365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.432231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3117.455282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3117.558636] Lustre: lustre-MDT0000: new disk, initializing [ 3117.662279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3121.999880] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3131.614116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.747388] Lustre: 109285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3131.988022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3135.650344] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3144.309457] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3150.117565] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3156.943028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3157.223088] Lustre: 111253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3163.449283] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3164.405947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3173.403314] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3178.902130] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3179.112462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3184.377300] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3184.740227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3190.246019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.253096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.263480] LustreError: Skipped 1 previous similar message [ 3190.268040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.276549] Lustre: Skipped 5 previous similar messages [ 3190.301618] Lustre: Skipped 9 previous similar messages [ 3193.720819] Lustre: server umount lustre-OST0000 complete [ 3212.255154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3212.546194] Lustre: server umount lustre-MDT0000 complete [ 3216.270678] LustreError: 110081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786977993 with bad export cookie 5439995506373140250 [ 3216.282489] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.283161] LustreError: 110081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3216.601632] Lustre: server umount lustre-MDT0001 complete [ 3224.842600] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:46:41 (1786978001) [ 3233.228842] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3235.594554] Key type lgssc unregistered [ 3235.826804] LNet: 113744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3235.839705] LNetError: 113744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3235.856319] LNet: Removed LNI 192.168.203.136@tcp [ 3236.484654] Key type .llcrypt unregistered [ 3236.486733] Key type ._llcrypt unregistered [ 3252.163425] Key type ._llcrypt registered [ 3252.165213] Key type .llcrypt registered [ 3252.246967] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 3267.471510] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3268.502163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3268.690720] alg: No test for adler32 (adler32-zlib) [ 3269.831393] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3270.193888] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3271.887244] Key type lgssc registered [ 3273.011046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3312.896722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3312.918397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.150957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3314.177776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3314.272738] Lustre: lustre-MDT0000: new disk, initializing [ 3314.358322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.371936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3318.038293] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3326.837212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.094897] Lustre: lustre-OST0000: new disk, initializing [ 3327.100198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3327.103683] Lustre: Skipped 1 previous similar message [ 3327.107155] Lustre: 117918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3327.205146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.225182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3328.244694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3328.283835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3333.873281] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3345.171530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.438769] Lustre: lustre-OST0001: new disk, initializing [ 3345.446825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3345.462289] Lustre: 118784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3345.552360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.138210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3352.144396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3352.177725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3352.492778] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3362.275721] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.296241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.305189] Lustre: Skipped 1 previous similar message [ 3366.749662] Lustre: server umount lustre-MDT0000 complete [ 3373.068207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3377.392622] LustreError: 117104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978154 with bad export cookie 779392873323011554 [ 3377.406230] LustreError: 117104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3377.408242] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.482973] Lustre: server umount lustre-OST0000 complete [ 3380.890846] Lustre: server umount lustre-OST0001 complete [ 3390.631504] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:49:26 (1786978166) [ 3417.716687] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:49:54 (1786978194) [ 3428.472948] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3453.749539] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:50:30 (1786978230) [ 3495.106978] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 3505.514783] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3545.749603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.993915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3546.039191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3546.147947] Lustre: lustre-MDT0000: new disk, initializing [ 3546.282951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.322540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3551.885641] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3562.740767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.859549] Lustre: 126535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.932067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3562.941608] Lustre: Skipped 1 previous similar message [ 3563.041910] Lustre: lustre-MDT0001: new disk, initializing [ 3563.182315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3563.260800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3563.275898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3569.068806] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3580.285465] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3587.233231] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3596.325877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3596.775801] Lustre: lustre-OST0000: new disk, initializing [ 3596.785214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3596.795714] Lustre: 128506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3596.912089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.573048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3603.588574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3603.710061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3604.303880] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3615.014521] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.721988] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3621.010079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3626.964937] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3627.273118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3634.664570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.666046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3634.680795] Lustre: Skipped 1 previous similar message [ 3634.684225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3634.708623] LustreError: Skipped 1 previous similar message [ 3636.122876] Lustre: server umount lustre-OST0000 complete [ 3644.897588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.905678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3644.912644] Lustre: Skipped 2 previous similar messages [ 3644.934235] Lustre: Skipped 2 previous similar messages [ 3646.849128] Lustre: server umount lustre-MDT0000 complete [ 3650.019139] LustreError: 126543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.039886] LustreError: 126543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3651.069429] LustreError: 127330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978428 with bad export cookie 779392873323012779 [ 3651.083933] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.085524] LustreError: 127330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.610416] Lustre: server umount lustre-MDT0001 complete [ 3661.597394] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:53:57 (1786978437) [ 3669.343116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.870879] LustreError: 130921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.915839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.514097] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3675.113848] LustreError: 130922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.203966] LustreError: 130921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.325714] LustreError: 130922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.401227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.579474] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3700.869868] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3707.715860] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3716.244049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.723233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.730146] Lustre: Skipped 1 previous similar message [ 3724.007651] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3732.117701] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3747.603466] Lustre: *** cfs_fail_loc=159, val=0*** [ 3763.008920] Lustre: lustre-MDT0000: Client 9c7ea0ae-e3c5-4665-9230-67626abe844b (at 192.168.203.36@tcp) reconnecting [ 3765.003226] Lustre: *** cfs_fail_loc=159, val=0*** [ 3765.010328] Lustre: Skipped 5 previous similar messages [ 3781.429597] Lustre: lustre-MDT0000: Client 9c7ea0ae-e3c5-4665-9230-67626abe844b (at 192.168.203.36@tcp) reconnecting [ 3785.708190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3785.713301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.785281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.744434] Lustre: server umount lustre-OST0000 complete [ 3793.376265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3793.382171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.397149] Lustre: Skipped 1 previous similar message [ 3793.402576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.404906] Lustre: Skipped 1 previous similar message [ 3798.400940] Lustre: server umount lustre-MDT0000 complete [ 3801.059268] LustreError: 130917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.086616] LustreError: 130917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3802.535468] LustreError: 130902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978579 with bad export cookie 779392873323013661 [ 3802.541989] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3802.556656] LustreError: 130902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3802.987416] Lustre: server umount lustre-MDT0001 complete [ 3814.202378] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 3817.030778] Key type lgssc unregistered [ 3817.360346] LNet: 134717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3817.366343] LNetError: 134717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3818.404937] LNet: Removed LNI 192.168.203.136@tcp [ 3819.296511] Key type .llcrypt unregistered [ 3819.301950] Key type ._llcrypt unregistered [ 3845.423903] Key type ._llcrypt registered [ 3845.426424] Key type .llcrypt registered [ 3845.534390] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 3846.906094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3846.939093] alg: No test for adler32 (adler32-zlib) [ 3848.130920] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 3848.422377] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 3850.111412] Key type lgssc registered [ 3851.202615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3860.812287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.829908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.355848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.504756] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3872.740793] LustreError: 136133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.767643] LustreError: 136133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3875.957532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.412314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3881.454080] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3889.033270] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3895.226194] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3903.069355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.396211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.506811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3909.433455] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 3917.272987] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3929.657559] Lustre: *** cfs_fail_loc=159, val=0*** [ 3945.240903] Lustre: lustre-MDT0000: Client c9399337-4188-409e-acd6-656327036ef5 (at 192.168.203.36@tcp) reconnecting [ 3949.080295] Lustre: *** cfs_fail_loc=159, val=0*** [ 3949.094864] Lustre: Skipped 1 previous similar message [ 3965.805723] Lustre: lustre-MDT0001: Client c9399337-4188-409e-acd6-656327036ef5 (at 192.168.203.36@tcp) reconnecting [ 3967.942759] Lustre: *** cfs_fail_loc=159, val=0*** [ 3967.946483] Lustre: Skipped 1 previous similar message [ 3984.233755] Lustre: lustre-MDT0001: Client c9399337-4188-409e-acd6-656327036ef5 (at 192.168.203.36@tcp) reconnecting [ 3989.558394] Lustre: 139178:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3992.970469] Lustre: *** cfs_fail_loc=159, val=0*** [ 3992.972564] Lustre: Skipped 4 previous similar messages [ 4009.260156] Lustre: lustre-MDT0000: Client f4af0c65-6ba6-48ff-b0cf-29ae08bf4146 (at 192.168.203.36@tcp) reconnecting [ 4011.045939] Lustre: *** cfs_fail_loc=159, val=0*** [ 4011.053587] Lustre: Skipped 21 previous similar messages [ 4027.714182] Lustre: lustre-MDT0000: Client f4af0c65-6ba6-48ff-b0cf-29ae08bf4146 (at 192.168.203.36@tcp) reconnecting [ 4029.435653] Lustre: 139471:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4035.553078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4035.560861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.580631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4036.086730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4036.087328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.094506] LustreError: Skipped 2 previous similar messages [ 4036.127087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.653879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4039.679818] Lustre: Skipped 1 previous similar message [ 4041.845417] Lustre: server umount lustre-OST0000 complete [ 4048.331715] Lustre: server umount lustre-MDT0000 complete [ 4049.888320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.901404] LustreError: 139295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.922704] Lustre: Skipped 1 previous similar message [ 4052.237304] LustreError: 136114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786978829 with bad export cookie 2058696515242434411 [ 4052.242624] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4052.262449] LustreError: 136114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4052.901757] Lustre: server umount lustre-MDT0001 complete [ 4064.850794] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4067.446777] Key type lgssc unregistered [ 4067.712071] LNet: 140526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4067.726827] LNetError: 140526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4067.737080] LNet: Removed LNI 192.168.203.136@tcp [ 4068.832600] Key type .llcrypt unregistered [ 4068.834380] Key type ._llcrypt unregistered [ 4094.393562] Key type ._llcrypt registered [ 4094.397674] Key type .llcrypt registered [ 4094.520442] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4095.646484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4095.879990] alg: No test for adler32 (adler32-zlib) [ 4096.916553] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4097.198704] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4098.887217] Key type lgssc registered [ 4100.126992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4108.467587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4108.490444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.085856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.051599] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4120.549943] LustreError: 141956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.567240] LustreError: 141956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4123.786582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.144191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4129.355902] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4138.080191] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4145.145308] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4153.624520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4154.045805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.085294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4161.557538] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4169.412096] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4170.645393] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4177.576425] Lustre: 144539:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4181.655363] Lustre: 144587:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4183.601735] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4184.639067] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4189.415621] Lustre: 144735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4190.647659] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4191.761456] Lustre: 144834:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4200.418974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4200.430923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.432873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.457135] Lustre: Skipped 1 previous similar message [ 4200.477586] Lustre: Skipped 1 previous similar message [ 4202.847934] Lustre: server umount lustre-OST0000 complete [ 4210.656434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.659587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.669655] Lustre: Skipped 1 previous similar message [ 4215.777242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.790499] Lustre: Skipped 2 previous similar messages [ 4220.898866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4220.909525] Lustre: Skipped 1 previous similar message [ 4221.407127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4221.594551] Lustre: server umount lustre-MDT0000 complete [ 4225.541651] LustreError: 141937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979002 with bad export cookie 3644199511667936085 [ 4225.552740] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4225.560274] LustreError: 141937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4226.074393] Lustre: server umount lustre-MDT0001 complete [ 4238.920637] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4241.958800] Key type lgssc unregistered [ 4242.291733] LNet: 145902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4242.304658] LNetError: 145902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4242.332875] LNet: Removed LNI 192.168.203.136@tcp [ 4243.158319] Key type .llcrypt unregistered [ 4243.160661] Key type ._llcrypt unregistered [ 4271.433962] Key type ._llcrypt registered [ 4271.436294] Key type .llcrypt registered [ 4271.523820] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4272.602764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4272.616976] alg: No test for adler32 (adler32-zlib) [ 4273.781803] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4274.123729] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4275.856000] Key type lgssc registered [ 4277.036751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4284.505791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.530839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.988798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.224395] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4295.138060] LustreError: 147318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.165425] LustreError: 147318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4299.495612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.936064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4304.785455] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4313.951620] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4320.594761] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4328.066031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.638300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.697979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4337.533193] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4345.026528] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4347.356855] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4354.134214] Lustre: *** cfs_fail_loc=159, val=0*** [ 4370.751996] Lustre: lustre-MDT0000: Client fdc4ebed-2c8c-4ef4-a30b-2e5e9a45c389 (at 192.168.203.36@tcp) reconnecting [ 4391.395834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.396116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4391.414606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4391.425837] Lustre: Skipped 1 previous similar message [ 4396.516338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4396.526674] Lustre: Skipped 1 previous similar message [ 4401.636668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4401.658139] Lustre: Skipped 1 previous similar message [ 4402.143169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4402.460620] Lustre: server umount lustre-OST0000 complete [ 4411.875603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.882983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.889929] Lustre: Skipped 2 previous similar messages [ 4413.773344] Lustre: server umount lustre-MDT0000 complete [ 4416.997576] LustreError: 149862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.012705] LustreError: 149862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4417.147668] LustreError: 147298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979194 with bad export cookie 16440828994100877046 [ 4417.149320] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.159515] LustreError: 147298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4417.443768] Lustre: server umount lustre-MDT0001 complete [ 4430.009677] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4433.593288] Key type lgssc unregistered [ 4434.109456] LNet: 150993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4434.119858] LNetError: 150993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4434.139238] LNet: Removed LNI 192.168.203.136@tcp [ 4435.452038] Key type .llcrypt unregistered [ 4435.453923] Key type ._llcrypt unregistered [ 4462.906438] Key type ._llcrypt registered [ 4462.909754] Key type .llcrypt registered [ 4463.082343] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4464.026825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4464.051611] alg: No test for adler32 (adler32-zlib) [ 4465.126986] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4465.323247] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4467.031869] Key type lgssc registered [ 4468.323527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4475.861969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4475.896373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.274871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.562613] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4487.665513] LustreError: 152408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.700087] LustreError: 152408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4490.437751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.908656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.757747] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4503.498288] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4509.506438] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4516.361143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.968178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.109408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4523.480815] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4530.931383] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4535.422876] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4556.227472] Lustre: lustre-MDT0001: evicting f308d94d-caec-4e70-9595-296e2fc31a72 (at 192.168.203.36@tcp) by administrative request [ 4556.235312] Lustre: Skipped 1 previous similar message [ 4586.981876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.982547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4586.992939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4592.098240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4592.114796] Lustre: Skipped 2 previous similar messages [ 4597.227732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.242166] Lustre: Skipped 1 previous similar message [ 4599.263139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.551909] Lustre: server umount lustre-OST0000 complete [ 4607.457200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.474766] Lustre: Skipped 1 previous similar message [ 4607.483586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4607.493542] Lustre: Skipped 1 previous similar message [ 4609.882485] Lustre: server umount lustre-MDT0000 complete [ 4612.583399] LustreError: 152408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.608810] LustreError: 152408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4613.224396] LustreError: 152388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979390 with bad export cookie 413282072133986064 [ 4613.249274] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4613.532830] Lustre: server umount lustre-MDT0001 complete [ 4624.462453] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4627.464624] Key type lgssc unregistered [ 4627.734387] LNet: 156483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4627.751485] LNetError: 156483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4627.788798] LNet: Removed LNI 192.168.203.136@tcp [ 4628.548579] Key type .llcrypt unregistered [ 4628.550492] Key type ._llcrypt unregistered [ 4701.522498] Key type ._llcrypt registered [ 4701.524802] Key type .llcrypt registered [ 4701.706037] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4703.273355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4703.323136] alg: No test for adler32 (adler32-zlib) [ 4704.480285] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4704.808741] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4706.543178] Key type lgssc registered [ 4707.884895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4716.893096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4716.929113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4718.514503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4723.817714] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4728.801834] LustreError: 158982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.857331] LustreError: 158982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4733.927910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.929063] LustreError: 158981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.523560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4740.409365] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4749.002728] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4755.865940] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4762.854154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4763.052027] LustreError: 160808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.365469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4768.756635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4769.154592] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4777.259033] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4780.038118] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4794.336449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4794.340358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4794.367684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.369380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4795.373508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.388024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.413174] Lustre: Skipped 1 previous similar message [ 4797.271252] Lustre: server umount lustre-OST0000 complete [ 4805.601543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.606252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4805.635149] Lustre: Skipped 1 previous similar message [ 4805.662546] Lustre: Skipped 1 previous similar message [ 4808.142378] Lustre: server umount lustre-MDT0000 complete [ 4810.731906] LustreError: 158981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.752251] LustreError: 158981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4812.644813] LustreError: 158961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979589 with bad export cookie 2619138134403109201 [ 4812.658845] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4813.371881] Lustre: server umount lustre-MDT0001 complete [ 4824.036199] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 4827.396240] Key type lgssc unregistered [ 4827.712497] LNet: 162548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4827.718391] LNetError: 162548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4827.740967] LNet: Removed LNI 192.168.203.136@tcp [ 4828.615584] Key type .llcrypt unregistered [ 4828.621028] Key type ._llcrypt unregistered [ 4854.954139] Key type ._llcrypt registered [ 4854.958791] Key type .llcrypt registered [ 4855.082355] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4856.369725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4856.491104] alg: No test for adler32 (adler32-zlib) [ 4857.602873] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 4857.865218] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 4859.536281] Key type lgssc registered [ 4860.732609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4876.907770] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:14:13 (1786979653) [ 4894.625953] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 4903.999208] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 4938.939784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4938.986360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.312206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4940.369968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4940.512150] Lustre: lustre-MDT0000: new disk, initializing [ 4940.659075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4940.681759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4945.633539] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4956.847837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.973360] Lustre: 166815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4957.033101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4957.036667] Lustre: Skipped 1 previous similar message [ 4957.041037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4957.106722] Lustre: lustre-MDT0001: new disk, initializing [ 4957.181725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.222783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4957.237148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4962.077731] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4972.353523] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4978.848459] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4986.007075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.379059] Lustre: lustre-OST0000: new disk, initializing [ 4986.391105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4986.397393] Lustre: 168784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4986.482410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4993.497989] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 4996.667545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4996.682693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4996.809579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5004.667157] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5017.573558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5017.588349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5017.595129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5017.605236] Lustre: Skipped 1 previous similar message [ 5017.638562] Lustre: Skipped 1 previous similar message [ 5021.276884] Lustre: server umount lustre-OST0000 complete [ 5027.811215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.830123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5027.836851] Lustre: Skipped 1 previous similar message [ 5031.467087] Lustre: server umount lustre-MDT0000 complete [ 5035.306933] LustreError: 166806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786979812 with bad export cookie 10999904548183737063 [ 5035.307653] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.327711] LustreError: 166806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5035.691351] Lustre: server umount lustre-MDT0001 complete [ 5046.633316] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 5049.450106] Key type lgssc unregistered [ 5049.701973] LNet: 170693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5049.710312] LNetError: 170693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5049.722172] LNet: Removed LNI 192.168.203.136@tcp [ 5050.526665] Key type .llcrypt unregistered [ 5050.529288] Key type ._llcrypt unregistered [ 5072.924729] Key type ._llcrypt registered [ 5072.927229] Key type .llcrypt registered [ 5073.006468] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 5073.783084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5073.813878] alg: No test for adler32 (adler32-zlib) [ 5074.930501] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 5075.244449] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 5076.919248] Key type lgssc registered [ 5078.000759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5085.232284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5085.245414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.686068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.817717] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 5096.928838] LustreError: 172113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.953127] LustreError: 172113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5099.493747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.927936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5105.091441] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 5113.937807] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5120.487837] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5126.606069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5126.938852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.058304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5132.471461] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 5139.439128] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5149.748835] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5149.766157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.778824] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5149.791471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.802717] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5150.739814] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5150.752774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.761065] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5150.766063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.778572] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5151.583599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.360772] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5152.368610] Lustre: Skipped 1 previous similar message [ 5152.382889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.392613] Lustre: Skipped 1 previous similar message [ 5152.398039] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5152.402758] Lustre: Skipped 1 previous similar message [ 5152.406719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.415691] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5152.422620] Lustre: Skipped 1 previous similar message [ 5153.182353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.949411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.674611] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5154.682520] Lustre: Skipped 2 previous similar messages [ 5154.715822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.730749] Lustre: Skipped 2 previous similar messages [ 5154.742355] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5154.748661] Lustre: Skipped 2 previous similar messages [ 5154.753289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.763387] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5154.769306] Lustre: Skipped 2 previous similar messages [ 5155.502404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.292443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.065370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.943748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.795407] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5158.803227] Lustre: Skipped 4 previous similar messages [ 5158.814481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.822436] Lustre: Skipped 4 previous similar messages [ 5158.825850] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5158.830508] Lustre: Skipped 4 previous similar messages [ 5158.833889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.852366] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5158.860342] Lustre: Skipped 4 previous similar messages [ 5159.738949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.485358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.401818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.375281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.302399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.040220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.032033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.858654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.795637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.685956] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5167.689802] Lustre: Skipped 9 previous similar messages [ 5167.713265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.739239] Lustre: Skipped 9 previous similar messages [ 5167.747455] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5167.752934] Lustre: Skipped 9 previous similar messages [ 5167.756078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.763878] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5167.768129] Lustre: Skipped 9 previous similar messages [ 5168.763268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.652659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.501619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.527458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.444883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.360377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.473574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.328044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.276459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.281286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.105766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.965183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.006264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.008649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.939649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.832260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.766558] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5183.776988] Lustre: Skipped 16 previous similar messages [ 5183.804844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.820075] Lustre: Skipped 16 previous similar messages [ 5183.825964] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5183.829261] Lustre: Skipped 16 previous similar messages [ 5183.833821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.845079] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5183.858976] Lustre: Skipped 16 previous similar messages [ 5184.666068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.565413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.335434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.116217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.975353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.844766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.646889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.696270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.620158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.547307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.458291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.568950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.474057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.479606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.258955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.187460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.994082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.839302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.655140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.326195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.084153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.884251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.646929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.606878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.376810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.182977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.989141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.749383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.491760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.375078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.299470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.383174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.461606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.413328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.607395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.690394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.681626] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5216.688176] Lustre: Skipped 36 previous similar messages [ 5216.706513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.717418] Lustre: Skipped 36 previous similar messages [ 5216.720335] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5216.722618] Lustre: Skipped 36 previous similar messages [ 5216.727828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.741215] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5216.748824] Lustre: Skipped 36 previous similar messages [ 5217.919242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.938069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.874763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.972399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.757852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.461568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.500352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.456851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.515591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.445754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.362451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.287410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.242732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.182289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.391099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.383986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.331715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.258657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.179881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.963404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.869232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.717359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.609752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.450257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.363872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.207129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.136447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.014844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.056468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.947057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.809575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.661027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.604916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.487133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.455693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.215149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.064187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.904466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.960908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.877174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.801138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.697509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.596190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.548592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.623874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.676730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.558319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.433130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.292034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.166427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.989051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.019239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.966075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.108971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.233537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.213701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.178594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.952122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.857552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.643911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.662671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.676865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.825820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.833403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.991909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.043451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.946323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.047228] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5281.063155] Lustre: Skipped 67 previous similar messages [ 5281.084738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.095692] Lustre: Skipped 67 previous similar messages [ 5281.100547] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5281.103731] Lustre: Skipped 67 previous similar messages [ 5281.108879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.135527] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5281.153033] Lustre: Skipped 67 previous similar messages [ 5282.184696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.128962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.011304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.964328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.836699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.793426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.638201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.538749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.402371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.304168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.124259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.887013] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.797405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.552628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.466301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.359685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.152200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.096954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.024958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.098112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.131283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.198961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.246553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.278888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.244754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.127585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.042622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.097317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.990316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.893754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.691657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.604063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.369771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.417765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.385294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.409881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.275882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.195973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.087112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.935326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.719883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.813971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.935558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.086179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.144366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.094903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.154658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.255127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.248803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.261089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.175573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.020023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.848572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.838493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.668228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.539946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.446898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.372784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.175314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.139025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.934822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.918819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.775067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.820835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.063824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.420768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.554821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.614656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.549096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.601356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.571578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.536116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.473129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.454637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.433346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.450701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.653056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.487065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.418049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.136534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.954807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.780260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.709976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.427195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.418621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.368538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.454921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.255105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.203358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.162716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.209567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.123717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.020407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.865157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.763411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.583994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.409272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.272635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.290252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.280918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.343045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.253199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.054825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.927653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.823944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.679861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.866899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.887579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.972654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.170949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.347644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.455791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.701560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.794056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.878623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.917991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.044672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.890359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.064866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.338247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.757224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.694534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.853737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.851949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.948459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.727254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.786094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.769651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.075261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.857210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.963919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.720781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.668241] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5409.674185] Lustre: Skipped 132 previous similar messages [ 5409.698786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.716106] Lustre: Skipped 132 previous similar messages [ 5409.725058] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5409.732761] Lustre: Skipped 132 previous similar messages [ 5409.736780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.748416] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5409.751618] Lustre: Skipped 132 previous similar messages [ 5410.698893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.738502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.804084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.108057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.312179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.632556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.845146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.115313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.547702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.699431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.839366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.938979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.101262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.107912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.935549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.819949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.910924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.882528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.218963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.189116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.368986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.469309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.492235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.427915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.307042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.335439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.395395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.477209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.438645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.342075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.153624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.130665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.192672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.502758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.554890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.667935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.969366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.894296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.999491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.809561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.905882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.850381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.991542] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5456.004800] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5456.022913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.091957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.199273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.590398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.619656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.793507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.646539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.649392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.741081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.663730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.791793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.637674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.658175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.483670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.407690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.321475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.415140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.577111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.682580] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5474.699588] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5474.716907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.546845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.459305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.406056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.240841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.148062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.049041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.011903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.032053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.836269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.778391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.838522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.943233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.982669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.065057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.118657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.215206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.310479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.399908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.318994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.251518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.250638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.241496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.300297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.076120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.969344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.897727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.829640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.784579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.763436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.833222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.842740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.985337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.061947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.259575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.410297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.387657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.439931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.582434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.431905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.398509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.332257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.286548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.412211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.561403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.488449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.498105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.311804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.226708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.162897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.929989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.957418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.842660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.775911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.674293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.663412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.496424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.511466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.676784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.853649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.615383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.600919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.542732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.472223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.535307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.712156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.729873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.796813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.847883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.846452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.793457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.716681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.592893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.459057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.294562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.169191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.997224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.827921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.680404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.525103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.395616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.185428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.947401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.948804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.749244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.762366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.697078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.563377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.435868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.203896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.048684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.853296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.597188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.314837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.024900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.804197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.490700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.287076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.132672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.066093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.090682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.039728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.950157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.822032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.657665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.503535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.360474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.197926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.142519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.100893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.179045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.059836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.035830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.916377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.913328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.984392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.874618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.706826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.616332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.631399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.594981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.492546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.391303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.238507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.335450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.280726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.235072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.210479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.203483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.273604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.307679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.372322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.640304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.728353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.650215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.679349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.540630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.484744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.502392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.624397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.535151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.518127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.471593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.374298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.445574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.531240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.638736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.571152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.669389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.750351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.676644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.563311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.544131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.499761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.410824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.329088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.275268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.323035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.589413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.797970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.873890] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5627.900968] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5627.907967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.103760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.183354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.260536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.211963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.305248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.254590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.196600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.045302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.980504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.914628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.952542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.882546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.816443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.903916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.974825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.982483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.022590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.064138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.113185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.296627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.270980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.495654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.600993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.672985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.762947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.685365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.478300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.272759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.262593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.077774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.853335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.715231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.665979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.531432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.421185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.185608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.987164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.847594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.806369] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5665.812023] Lustre: Skipped 259 previous similar messages [ 5665.841469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.853546] Lustre: Skipped 259 previous similar messages [ 5665.863637] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5665.873347] Lustre: Skipped 259 previous similar messages [ 5665.884228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.894975] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5665.903614] Lustre: Skipped 259 previous similar messages [ 5666.716511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.506817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.344673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.330967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.206481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.102720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.932527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.859359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.906147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.978662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.965592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.837059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.774313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.616851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.409571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.248533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.156873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.848554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.738616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.473253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.306184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.082414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.969940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.704978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.703557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.688133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.610623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.501176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.321942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.185315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.156153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.315301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.229262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.125398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.209433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.197421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.184062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.304444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.221080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.965726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.970946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.123649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.170804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.326044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.772861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.997977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.089817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.090796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.282427] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5712.299357] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5712.306539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.288355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.531956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.693856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.253638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.813545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.223634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.689339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.999813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.674820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.343100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.980684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.369335] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5729.397606] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5729.405575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.013931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.438791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.399247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.325963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.156649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.248261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.188701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.353421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.287356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.354406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.465674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.436567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.520557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.515503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.760818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.719748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.964149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.013436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.047435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.196033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.105615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.227766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.278858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.358225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.406739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.220939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.602835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.766307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.834345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.700603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.666546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.593591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.979294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.983626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.048703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.073868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.041540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.339616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.737292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.100915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.008171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.929900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.198923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.985544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.084737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.892774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.960041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.968318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.224567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.547685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.555266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.621836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.833058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.993779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.092839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.435458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.772473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.713075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.467152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.152682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.926825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.775476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.633605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.421981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.221547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.069096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.742412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.377349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.175876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.032675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.753785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.530632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.501252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.337835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.146412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.834724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.665930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.373933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.214565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.940543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.022283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.854315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.720593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.729369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.613104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.701439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.552428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.775371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.998840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.063066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.126278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.067281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.988032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.066619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.133513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.191664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.266583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.341734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.431548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.514442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.372656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.453433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.305716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.286327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.169277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.147709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.128624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.889129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.776774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.746729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.582357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.621829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.663825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.737802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.691989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.614352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.537073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.584450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.571134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.492532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.196186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.055660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.771164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.672067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.516542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.446940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.297833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.257983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.276461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.211517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.048497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.013903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.895950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.899263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.815433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.714476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.501465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.498146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.373504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.433144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.329169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.286818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.144428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.127299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.079743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.976437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.971909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.871906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.796841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.801363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.846884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.799336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.011160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.206161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.300085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.200607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.180106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.999164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.958780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.987319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.905337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.834434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.752163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.732269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.665678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.647976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.627080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.566741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.719990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.772802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.676227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.596771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.529237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.402452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.218303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.119625] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5902.128374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.148188] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5903.014210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.975840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.946744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.985801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.954263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.029615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.827674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.697152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.627081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.396133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.315945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.329016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.301139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.318756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.300127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.323578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.607567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.841614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.783988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.751669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.906551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.192293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.116137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.295359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.358837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.330384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.331378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.430980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.472757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.500529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.655080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.587912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.718187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.739158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.704145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.626027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.577304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.268996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.942247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.631840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.553612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.532399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.595475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.629316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.601364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.521811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.570412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.487527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.362653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.347892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.272234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.320430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.247556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.229259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.252343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.121296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.163236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.267300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.411452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.410619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.283673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.098838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.146197] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5964.164416] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5964.181724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.390046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.436515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.590825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.790543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.792635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.713423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.605779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.537768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.307874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.201734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.112598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.005722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.878281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.852677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.744225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.569806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.411224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.371313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.448186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.404173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.367711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.259124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.164813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.370612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.359932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.547830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.606295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.762542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.743083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.853396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.822561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.810643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.808732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.227135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.301932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.426626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.339679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.195052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.959823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.962656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.946500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.907397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.072222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.261295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.342358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.389818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.270566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.262617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.209250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.205178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.505514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.399985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.359370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.221575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.342857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.247972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.578966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.559402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.873754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.408437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.687479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.644968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.608372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.750195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.930084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.334736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.507611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.591917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.922770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.793246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.883187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.083767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.109569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.235402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.285969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.272415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.406218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.461887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.596922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.549256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.590156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.503063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.476499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.466631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.619637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.740946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.150227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.201697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.275391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.339063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.280810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.679525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.606935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.947305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.132958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.444570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.939609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.328950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.577495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.776830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.993353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.785597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.184219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.532085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.882936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.031602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.266874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.704721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.895282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.091115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.285200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.355958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.824868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.168614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.240748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.090308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.190822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.202930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.205646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.308854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.535482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.586598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.437767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.600842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.536268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.593704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.680812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.516059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.654729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.802990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.682040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.803234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.790712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.863313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.886975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.060703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.038285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.030980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.085029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.912182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.871475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.677221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.572934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.434072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.458194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.657821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.748682] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6121.759904] LustreError: 172132:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6121.765542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.911296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.809485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.971064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.799601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.814067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.686908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.922161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.806442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.832562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.725813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.763240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.594377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.516667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.691557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.875698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.071351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.059621] LustreError: 174203:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6591.072579] LustreError: 172132:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7313.379470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.387331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.410500] Lustre: Skipped 463 previous similar messages [ 7313.431954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.424581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.434212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7315.437175] Lustre: Skipped 1 previous similar message [ 7319.684450] Lustre: server umount lustre-OST0000 complete [ 7325.672801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.690729] Lustre: Skipped 1 previous similar message [ 7329.502226] Lustre: server umount lustre-MDT0000 complete [ 7330.785275] LustreError: 172113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7330.806751] LustreError: 172113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7333.266967] LustreError: 172093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982110 with bad export cookie 2047962968163129932 [ 7333.268938] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7333.277050] LustreError: 172093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7333.748533] Lustre: server umount lustre-MDT0001 complete [ 7346.018070] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 7348.839021] Key type lgssc unregistered [ 7349.169267] LNet: 227394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7349.197475] LNetError: 227394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7349.238800] LNet: Removed LNI 192.168.203.136@tcp [ 7350.626288] Key type .llcrypt unregistered [ 7350.629677] Key type ._llcrypt unregistered [ 7378.005738] Key type ._llcrypt registered [ 7378.011860] Key type .llcrypt registered [ 7378.186754] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 7379.695856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7379.786267] alg: No test for adler32 (adler32-zlib) [ 7381.189419] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 7381.581979] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 7383.343206] Key type lgssc registered [ 7384.949424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7395.735947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7395.776984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.317671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7402.866480] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7407.588277] LustreError: 228810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.614363] LustreError: 228810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7412.709543] LustreError: 228811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.783288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.021298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7419.972468] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7429.063456] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7436.111836] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7444.646529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.119472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.175725] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7462.280285] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7467.676850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7471.455941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7473.424249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.438117] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7473.453907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7473.474818] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7473.476298] LustreError: 228830:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7478.245051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7478.255366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7478.274958] Lustre: Skipped 1 previous similar message [ 7478.302680] Lustre: Skipped 1 previous similar message [ 7481.902249] Lustre: server umount lustre-OST0000 complete [ 7493.551138] LustreError: 228830:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7516.843842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.159973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.178105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7518.201168] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7518.205296] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7518.214766] Lustre: Skipped 1 previous similar message [ 7522.985573] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7530.827577] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7532.434237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7538.403666] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0254383b800.ost_server_uuid 50 [ 7540.084593] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0254383b800.ost_server_uuid in FULL state after 0 sec [ 7545.315380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.325184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.340049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.348459] Lustre: Skipped 1 previous similar message [ 7548.897477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7548.901765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7548.910598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.072904] Lustre: server umount lustre-OST0000 complete [ 7557.602948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7557.616160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.629386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7557.640482] Lustre: Skipped 1 previous similar message [ 7560.983517] Lustre: server umount lustre-MDT0000 complete [ 7564.256986] LustreError: 228807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.273554] LustreError: 228807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7565.020368] LustreError: 228791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982341 with bad export cookie 5669348831580641337 [ 7565.024787] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.029805] LustreError: 228791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7565.465973] Lustre: server umount lustre-MDT0001 complete [ 7578.926427] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 7582.845652] Key type lgssc unregistered [ 7583.297865] LNet: 234021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7583.304155] LNetError: 234021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7583.320209] LNet: Removed LNI 192.168.203.136@tcp [ 7584.442558] Key type .llcrypt unregistered [ 7584.446373] Key type ._llcrypt unregistered [ 7619.074159] Key type ._llcrypt registered [ 7619.076308] Key type .llcrypt registered [ 7619.175709] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 7620.552496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7620.659730] alg: No test for adler32 (adler32-zlib) [ 7621.782111] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 7622.039970] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 7623.719960] Key type lgssc registered [ 7625.407394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7635.767902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7635.795838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.250017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7641.822416] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7647.723170] LustreError: 235753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.747732] LustreError: 235753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7651.312127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.716949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.169285] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7660.647061] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7669.213856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7669.482975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.068266] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7682.809805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7685.414715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7685.688403] Lustre: lustre-OST0001: new disk, initializing [ 7685.706280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7685.720327] Lustre: 238072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7685.851683] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.012396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7691.020459] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7691.190940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7693.275644] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7705.061294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7714.630825] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7732.194663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.217663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.321118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.345207] Lustre: Skipped 6 previous similar messages [ 7737.945828] Lustre: server umount lustre-MDT0000 complete [ 7741.991575] LustreError: 239018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982518 with bad export cookie 1442748779753324768 [ 7742.006382] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7742.014600] LustreError: 239018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7742.381165] Lustre: server umount lustre-MDT0001 complete [ 7757.080162] Lustre: server umount lustre-OST0000 complete [ 7758.822928] Lustre: 235016:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786982519/real 1786982519] req@ffff964c037ddf80 x1873786855064064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786982535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7758.856153] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.865119] Lustre: Skipped 3 previous similar messages [ 7760.747501] Lustre: server umount lustre-OST0001 complete [ 7780.703236] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 7789.822569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.172523] LustreError: 241183:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7790.249167] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.473734] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7795.689202] LustreError: 241184:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7800.800940] LustreError: 241183:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.994592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.285932] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7808.283544] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7811.678602] Lustre: 242325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7819.450402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.887968] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.994564] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7828.084430] LustreError: 242680:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.098399] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7834.404925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7835.726260] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7835.743417] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7841.327644] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7850.789691] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7854.473456] Lustre: 244198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7864.804938] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7864.823331] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.842778] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.647536] Lustre: server umount mylustre-MDT0000 complete [ 7871.460124] LustreError: 241178:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.472767] LustreError: 241178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7873.651734] LustreError: 241158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982650 with bad export cookie 1442748779753327631 [ 7873.656593] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7873.665922] LustreError: 241158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7873.973455] Lustre: server umount mylustre-MDT0001 complete [ 7887.999984] Lustre: server umount mylustre-OST0000 complete [ 7900.332844] Lustre: server umount mylustre-OST0001 complete [ 7918.884881] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 7929.329926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.796171] LustreError: 246505:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.902773] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.910713] Lustre: Skipped 1 previous similar message [ 7934.885833] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7945.165885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7950.003607] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7953.117759] Lustre: 247648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7960.287901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7968.382888] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7971.047110] LustreError: 248004:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.062895] LustreError: 248004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7971.065962] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7974.146465] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7976.348220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7982.089794] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 7982.881813] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 7989.709111] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7993.116174] Lustre: 249522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8002.020360] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8002.028512] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.056977] Lustre: Skipped 3 previous similar messages [ 8002.073185] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8002.084544] Lustre: Skipped 4 previous similar messages [ 8005.490950] Lustre: server umount tfs-MDT0000 complete [ 8008.586763] LustreError: 247650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982785 with bad export cookie 1442748779753329808 [ 8008.591862] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8008.599732] LustreError: 247650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8008.962086] Lustre: server umount tfs-MDT0001 complete [ 8023.130284] Lustre: server umount tfs-OST0000 complete [ 8037.459460] Lustre: server umount tfs-OST0001 complete [ 8058.170904] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 8067.499240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.967406] LustreError: 251772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8067.989977] LustreError: 251772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8068.043159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.048467] Lustre: Skipped 3 previous similar messages [ 8072.113524] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8079.621965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8084.338757] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8087.319197] Lustre: 252917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8093.666429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8099.503794] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8104.314050] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 8108.075209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8112.513360] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97) [ 8112.523347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8114.734526] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8122.442482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8126.475342] Lustre: 254787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8132.845652] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:08:29 (1786982909) [ 8148.464675] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8148.469556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8148.477148] Lustre: Skipped 4 previous similar messages [ 8148.497084] Lustre: Skipped 6 previous similar messages [ 8154.778421] Lustre: server umount lustre-MDT0000 complete [ 8157.816161] LustreError: 251749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786982934 with bad export cookie 1442748779753331726 [ 8157.827444] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.832357] LustreError: 251749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8158.697736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.707935] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8158.718062] Lustre: Skipped 3 previous similar messages [ 8158.724155] Lustre: Skipped 4 previous similar messages [ 8164.160747] Lustre: server umount lustre-MDT0001 complete [ 8173.799424] Lustre: server umount lustre-OST0000 complete [ 8183.216257] Lustre: server umount lustre-OST0001 complete [ 8188.741911] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 8198.683555] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 8231.520528] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8231.520528] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8231.520528] [ 8231.545926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8231.729955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8231.789482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8231.917238] Lustre: lustre-MDT0000: new disk, initializing [ 8232.024928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8232.031283] Lustre: Skipped 3 previous similar messages [ 8232.050617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8236.101084] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8245.924211] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8245.924211] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8245.924211] [ 8245.945975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8246.056702] Lustre: 257970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8246.139532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8246.149088] Lustre: Skipped 1 previous similar message [ 8246.222973] Lustre: lustre-MDT0001: new disk, initializing [ 8246.356962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8246.370313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8250.228473] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8258.969623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8259.215122] Lustre: lustre-OST0000: new disk, initializing [ 8259.219664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8259.233847] Lustre: 259553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8260.989615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8261.000162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8261.111589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8264.340141] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8274.911363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8275.109699] Lustre: lustre-OST0001: new disk, initializing [ 8275.112453] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8275.117473] Lustre: 260425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8277.120293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8277.132079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8277.220263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8280.370666] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8289.962456] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:11:06 (1786983066) [ 8302.045611] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:11:18 (1786983078) [ 8317.411231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8317.418483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.421637] LustreError: Skipped 1 previous similar message [ 8317.433496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8317.441930] Lustre: Skipped 3 previous similar messages [ 8321.703594] Lustre: server umount lustre-OST0000 complete [ 8323.561389] LustreError: 259538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.577938] LustreError: 259538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8328.160537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8330.955751] Lustre: server umount lustre-MDT0000 complete [ 8333.857546] LustreError: 257960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983110 with bad export cookie 1442748779753333504 [ 8333.859096] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8333.866877] LustreError: 257960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8340.251406] Lustre: server umount lustre-MDT0001 complete [ 8349.752575] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 8425.407512] Lustre: server umount lustre-OST0001 complete [ 8431.315513] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 8445.295388] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 8476.373316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.646492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8476.672400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8476.740259] Lustre: lustre-MDT0000: new disk, initializing [ 8476.822426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8480.796074] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8491.247369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.382675] Lustre: 265219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8491.719593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.732126] Lustre: Skipped 4 previous similar messages [ 8491.778943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8496.901103] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8506.133450] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8512.232388] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8518.353645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8518.637905] Lustre: lustre-OST0000: new disk, initializing [ 8518.640531] Lustre: Skipped 1 previous similar message [ 8518.643870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8518.648484] Lustre: Skipped 2 previous similar messages [ 8518.660486] Lustre: 267186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8520.559689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8520.567140] Lustre: Skipped 1 previous similar message [ 8520.574789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8520.672732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8524.524882] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8533.669883] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8541.159176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.179199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.183888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8541.198217] Lustre: Skipped 6 previous similar messages [ 8541.214271] Lustre: Skipped 7 previous similar messages [ 8546.598600] Lustre: server umount lustre-OST0000 complete [ 8556.651851] Lustre: server umount lustre-MDT0000 complete [ 8560.272262] LustreError: 265210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983337 with bad export cookie 1442748779753334974 [ 8560.293023] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8560.296121] LustreError: 265210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8560.650640] Lustre: server umount lustre-MDT0001 complete [ 8571.135438] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 8573.770583] Key type lgssc unregistered [ 8574.063752] LNet: 268948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8574.066579] LNetError: 268948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8574.097247] LNet: Removed LNI 192.168.203.136@tcp [ 8574.688156] Key type .llcrypt unregistered [ 8574.691919] Key type ._llcrypt unregistered [ 8597.079756] Key type ._llcrypt registered [ 8597.083130] Key type .llcrypt registered [ 8597.172361] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 8597.909991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8597.980687] alg: No test for adler32 (adler32-zlib) [ 8599.033662] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8599.337006] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 8601.041636] Key type lgssc registered [ 8601.979799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8608.909115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8608.925752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.322523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8614.142512] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8620.515560] LustreError: 270391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8620.529134] LustreError: 270391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8621.881440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.320544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8627.434379] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8635.295973] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8640.864858] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8647.675375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8648.077239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8653.618275] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8655.302791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8660.410518] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8661.220747] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8665.569630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8665.576425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.595590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8667.682494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.739767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.757070] Lustre: Skipped 1 previous similar message [ 8677.343882] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8677.587341] Lustre: server umount lustre-OST0000 complete [ 8682.978480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.980171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8682.986316] Lustre: Skipped 1 previous similar message [ 8683.000809] Lustre: Skipped 1 previous similar message [ 8688.097129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8688.100768] Lustre: Skipped 1 previous similar message [ 8695.264717] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8695.402168] Lustre: server umount lustre-MDT0000 complete [ 8698.339034] LustreError: 272685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.358703] LustreError: 272685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8699.099653] LustreError: 272921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983476 with bad export cookie 15789211656011580067 [ 8699.104901] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8699.111206] LustreError: 272921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8699.511437] Lustre: server umount lustre-MDT0001 complete [ 8709.690242] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 8711.929948] Key type lgssc unregistered [ 8712.214599] LNet: 273980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8712.219978] LNetError: 273980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8712.232159] LNet: Removed LNI 192.168.203.136@tcp [ 8713.023597] Key type .llcrypt unregistered [ 8713.029135] Key type ._llcrypt unregistered [ 8728.026205] Key type ._llcrypt registered [ 8728.033381] Key type .llcrypt registered [ 8728.165104] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 8729.601221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8729.768496] alg: No test for adler32 (adler32-zlib) [ 8730.951741] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8731.221592] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 8732.887175] Key type lgssc registered [ 8734.052244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8744.253737] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8744.294037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8759.967561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8765.088585] LustreError: 275174:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.136@tcp: failed processing log, type 4: rc = -110 [ 8791.776310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.079397] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 8804.004584] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8806.104670] Lustre: server umount lustre-OST0000 complete [ 8822.100980] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 8824.803170] Key type lgssc unregistered [ 8825.085601] LNet: 276809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8825.097528] LNetError: 276809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8825.115890] LNet: Removed LNI 192.168.203.136@tcp [ 8825.895337] Key type .llcrypt unregistered [ 8825.904513] Key type ._llcrypt unregistered [ 8870.939175] Key type ._llcrypt registered [ 8870.940746] Key type .llcrypt registered [ 8871.101622] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 8872.213704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8872.420404] alg: No test for adler32 (adler32-zlib) [ 8873.434487] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 8874.067356] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 8875.815439] Key type lgssc registered [ 8877.253375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8889.681437] loop0: detected capacity change from 0 to 204800000 [ 8889.900411] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8889.928736] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8890.278504] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8891.124481] loop0: detected capacity change from 0 to 204800000 [ 8891.322264] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8891.348800] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8891.658460] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8892.668969] loop0: detected capacity change from 0 to 204800000 [ 8892.870096] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8892.888622] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8893.041547] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8894.143293] loop0: detected capacity change from 0 to 204800000 [ 8894.298131] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8895.432822] loop0: detected capacity change from 0 to 204800000 [ 8895.460605] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8899.314726] loop0: detected capacity change from 0 to 204800000 [ 8899.367084] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8903.331986] loop0: detected capacity change from 0 to 204800000 [ 8903.378063] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8908.433136] loop0: detected capacity change from 0 to 204800000 [ 8908.478217] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8913.956403] loop0: detected capacity change from 0 to 204800000 [ 8914.080587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8914.149133] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.296266] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8914.312044] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8914.356866] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8914.397851] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8914.413872] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8914.443420] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8914.476437] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8914.508138] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8914.528302] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8914.575896] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8914.606143] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8914.642884] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8914.655475] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8914.671655] Lustre: 280223:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8921.467140] Lustre: server umount MGS complete [ 8922.495412] loop0: detected capacity change from 0 to 204800000 [ 8922.572778] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.964109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8923.016434] Lustre: lustre-MDT0000: Aborting client recovery [ 8924.104070] loop1: detected capacity change from 0 to 204800000 [ 8924.343345] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.446578] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8924.455879] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8924.465696] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8924.477099] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8924.502300] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8924.522420] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8924.545369] Lustre: 280655:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8924.566045] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8924.695784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8924.741444] Lustre: lustre-MDT0001: Aborting client recovery [ 8925.466137] loop2: detected capacity change from 0 to 204800000 [ 8925.644073] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.710783] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8925.718617] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8925.727983] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8925.735921] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8925.745927] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8925.754717] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8925.762805] Lustre: 280737:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8925.860570] Lustre: Found index 0 for lustre-OST0000, updating log [ 8925.894392] Lustre: 280755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8925.960780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8925.987872] Lustre: lustre-OST0000: Aborting recovery [ 8927.217860] loop3: detected capacity change from 0 to 204800000 [ 8927.506322] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8927.585447] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8927.595416] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8927.605689] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8927.620810] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8927.643728] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8927.664953] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8927.679546] Lustre: 280821:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8927.701642] Lustre: Found index 1 for lustre-OST0001, updating log [ 8927.742449] Lustre: 280826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8927.858295] Lustre: lustre-OST0001: Aborting recovery [ 8932.985520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8932.988852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8933.024439] LustreError: 280599:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8933.052664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8933.074799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8935.013410] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8935.032628] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8936.912169] LustreError: 280566:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 605us: rc = -2 [ 8937.501045] LustreError: 280567:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 580us: rc = -2 [ 8937.520913] LustreError: 280567:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 8950.239910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8950.248425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8950.263740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8950.752113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8950.772575] Lustre: Skipped 2 previous similar messages [ 8950.779930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8950.784929] Lustre: Skipped 2 previous similar messages [ 8955.878975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8961.003311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8961.014713] Lustre: Skipped 6 previous similar messages [ 8964.068275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8964.651721] Lustre: server umount lustre-MDT0000 complete [ 8965.572935] LustreError: 280548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786983742 with bad export cookie 8286414066335648764 [ 8965.580909] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.586172] LustreError: 280548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8966.122705] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.124543] LustreError: 281057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8966.128500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8966.142185] Lustre: Skipped 1 previous similar message [ 8966.153524] LustreError: 281057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8970.151077] LustreError: 280565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8970.163963] LustreError: 280565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8971.945238] Lustre: server umount lustre-MDT0001 complete [ 8979.105712] Lustre: server umount lustre-OST0000 complete [ 8986.396603] Lustre: server umount lustre-OST0001 complete [ 8992.694151] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 12:22:49 (1786983769) [ 9022.765213] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 9037.769135] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 9078.296260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.750774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.800803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9078.891524] Lustre: lustre-MDT0000: new disk, initializing [ 9079.016465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9079.028599] Lustre: Skipped 1 previous similar message [ 9079.051956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9083.470318] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9093.450537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.596809] Lustre: 284911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9093.646623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9093.650848] Lustre: Skipped 1 previous similar message [ 9093.714873] Lustre: lustre-MDT0001: new disk, initializing [ 9093.801849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9093.851900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9093.873170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9098.314227] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9108.544380] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9114.357567] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9121.412141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9121.747211] Lustre: lustre-OST0000: new disk, initializing [ 9121.757395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9121.768482] Lustre: 286880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9121.872752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9128.493808] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9129.050992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9129.062211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9129.211640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9138.043362] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9143.007548] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9155.189284] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9165.584928] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9176.838573] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9231.843327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9231.846695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.855737] LustreError: Skipped 1 previous similar message [ 9231.856664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9231.856669] Lustre: Skipped 3 previous similar messages [ 9231.888737] Lustre: Skipped 1 previous similar message [ 9234.646496] Lustre: server umount lustre-OST0000 complete [ 9242.082568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9251.298908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9251.309569] Lustre: Skipped 6 previous similar messages [ 9254.367126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9254.682663] Lustre: server umount lustre-MDT0000 complete [ 9256.417180] LustreError: 284918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9256.441178] LustreError: 284918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9258.581665] LustreError: 285716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984035 with bad export cookie 8286414066335651466 [ 9258.586597] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9258.588855] LustreError: 285716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9259.084842] Lustre: server umount lustre-MDT0001 complete [ 9267.151870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.587536] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9276.182473] Lustre: server umount MGS complete [ 9287.874789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.355268] LustreError: 290815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9288.376587] LustreError: 290828:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9288.446292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9293.655980] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9293.795582] LustreError: 290816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9303.729123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.047344] LustreError: 290815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.068409] LustreError: 290815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9310.049524] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9321.395968] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9329.042495] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9338.360830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9338.955266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.958804] Lustre: Skipped 1 previous similar message [ 9347.141680] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9356.131028] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9394.659694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.661506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.663890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.696753] Lustre: Skipped 2 previous similar messages [ 9400.559931] Lustre: server umount lustre-OST0000 complete [ 9411.156050] Lustre: server umount lustre-MDT0000 complete [ 9415.147355] LustreError: 290810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9415.178533] LustreError: 290810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9415.906628] LustreError: 293546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984192 with bad export cookie 8286414066335653419 [ 9415.919778] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.920123] LustreError: 293546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9416.726853] Lustre: server umount lustre-MDT0001 complete [ 9428.924318] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 9431.557484] Key type lgssc unregistered [ 9431.789320] LNet: 294769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9431.796881] LNetError: 294769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9431.812307] LNet: Removed LNI 192.168.203.136@tcp [ 9432.538122] Key type .llcrypt unregistered [ 9432.540213] Key type ._llcrypt unregistered [ 9468.845848] Key type ._llcrypt registered [ 9468.856671] Key type .llcrypt registered [ 9468.981176] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 9487.007740] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 9488.218544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9488.238668] alg: No test for adler32 (adler32-zlib) [ 9489.383916] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 9489.727508] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 9491.463167] Key type lgssc registered [ 9493.167870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9536.497683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9536.530554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9537.867187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9537.897353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9537.999167] Lustre: lustre-MDT0000: new disk, initializing [ 9538.066834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.082522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9543.054877] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9554.280083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.404162] Lustre: 298727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9554.458846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9554.474275] Lustre: Skipped 1 previous similar message [ 9554.575684] Lustre: lustre-MDT0001: new disk, initializing [ 9554.722892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9554.765505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9554.782748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9559.211942] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9569.367024] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9576.187556] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9582.773440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9583.169346] Lustre: lustre-OST0000: new disk, initializing [ 9583.172337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9583.177807] Lustre: 300699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9583.257555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9589.205635] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9593.438701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9593.455058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9593.570111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9598.721448] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9606.451357] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9621.393864] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9627.591849] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9675.751995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.756612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9675.761374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9675.767419] Lustre: Skipped 1 previous similar message [ 9675.790093] Lustre: Skipped 1 previous similar message [ 9680.874574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9681.237536] Lustre: server umount lustre-OST0000 complete [ 9685.984550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9685.989693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.996350] Lustre: Skipped 1 previous similar message [ 9686.029114] Lustre: Skipped 2 previous similar messages [ 9691.104886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9691.113206] Lustre: Skipped 1 previous similar message [ 9691.908078] Lustre: server umount lustre-MDT0000 complete [ 9695.771898] LustreError: 298720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984472 with bad export cookie 14585637700978183706 [ 9695.773202] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9695.800493] LustreError: 298720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9696.277553] Lustre: server umount lustre-MDT0001 complete [ 9706.122420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9711.908492] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9718.327191] Lustre: server umount MGS complete [ 9730.916559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.387523] LustreError: 304673:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9731.481215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9736.679670] LustreError: 304660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.706866] LustreError: 304660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9736.708540] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9741.805283] LustreError: 304661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.353495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9746.698066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9751.268732] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9759.867857] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9766.419846] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9774.239828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9774.743398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9782.119824] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9791.387309] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9827.302103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9827.317739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9827.317934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9827.325707] LustreError: Skipped 1 previous similar message [ 9827.334152] Lustre: Skipped 1 previous similar message [ 9830.433714] Lustre: server umount lustre-OST0000 complete [ 9837.536936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.540188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9837.563592] Lustre: Skipped 2 previous similar messages [ 9837.588511] Lustre: Skipped 1 previous similar message [ 9840.875392] Lustre: server umount lustre-MDT0000 complete [ 9842.664161] LustreError: 304656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.698095] LustreError: 304656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9845.896967] LustreError: 304641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786984622 with bad export cookie 14585637700978185561 [ 9845.900230] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9845.908607] LustreError: 304641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9846.392980] Lustre: server umount lustre-MDT0001 complete [ 9858.751087] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [ 9861.707440] Key type lgssc unregistered [ 9862.017555] LNet: 308614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9862.023810] LNetError: 308614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9862.037309] LNet: Removed LNI 192.168.203.136@tcp [ 9862.933686] Key type .llcrypt unregistered [ 9862.935640] Key type ._llcrypt unregistered [ 9884.916518] Key type ._llcrypt registered [ 9884.918812] Key type .llcrypt registered [ 9885.009926] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [ 9897.253413] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 9898.240230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9898.251029] alg: No test for adler32 (adler32-zlib) [ 9899.268750] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 9899.535104] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [ 9901.247289] Key type lgssc registered [ 9902.344534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9926.676742] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [ 9942.557718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9942.613294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.914391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9943.959865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9944.045323] Lustre: lustre-MDT0000: new disk, initializing [ 9944.136962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9944.186897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9949.120524] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9960.279180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9960.579493] Lustre: lustre-OST0000: new disk, initializing [ 9960.584916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9960.594590] Lustre: Skipped 1 previous similar message [ 9960.614775] Lustre: 312645:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9960.703980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9967.151081] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [ 9970.848659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9970.862589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9970.903499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10937.745305] Lustre: server umount lustre-MDT0000 complete [10941.520763] LustreError: 311830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985718 with bad export cookie 16843663988775464920 [10941.539697] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10951.362547] Lustre: server umount lustre-OST0000 complete [10978.201563] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [10988.527845] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [11028.357901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11028.584459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11028.619911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11028.676850] Lustre: lustre-MDT0000: new disk, initializing [11028.747792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11028.768670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11033.953991] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11045.096053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.226601] Lustre: 317001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11045.288262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11045.292871] Lustre: Skipped 1 previous similar message [11045.387762] Lustre: lustre-MDT0001: new disk, initializing [11045.495962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11045.570151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11045.605766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11049.982490] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11059.642594] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11066.428290] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11074.941416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11075.262990] Lustre: lustre-OST0000: new disk, initializing [11075.268780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11075.287721] Lustre: 318970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11075.368141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11082.883579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11082.906262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11082.955857] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11083.081812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11093.725355] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11099.285059] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11099.650379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11105.051487] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11105.321946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11108.832710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11108.853711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11108.890259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11113.957839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11113.967789] Lustre: Skipped 1 previous similar message [11119.076931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11119.084215] Lustre: Skipped 1 previous similar message [11122.143312] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11122.390954] Lustre: server umount lustre-OST0000 complete [11127.265065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11127.278643] LustreError: Skipped 1 previous similar message [11127.281964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.297235] Lustre: Skipped 1 previous similar message [11127.307462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11127.322895] Lustre: Skipped 1 previous similar message [11129.316175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11133.315668] Lustre: server umount lustre-MDT0000 complete [11137.521250] LustreError: 316992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786985914 with bad export cookie 16843663988778405739 [11137.531476] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11137.538974] LustreError: 316992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11137.990392] Lustre: server umount lustre-MDT0001 complete [11141.490285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11143.610403] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:58:39 (1786985919) [11150.238686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11150.611056] LustreError: 321284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.641655] LustreError: 321284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11150.682318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11155.710689] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11155.940483] LustreError: 321285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.068078] LustreError: 321284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.167114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.593964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11171.085760] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11179.656707] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11186.976732] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11194.901097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11195.224394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11201.939832] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11210.698622] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11219.778574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11220.074082] Lustre: lustre-OST0001: new disk, initializing [11220.084886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11220.108531] Lustre: 324120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11220.367901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11228.812475] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11228.818885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11228.845777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11228.888849] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11229.938641] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11229.957668] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11231.020827] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11231.036103] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11233.059678] LustreError: 324485:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11233.070960] LustreError: 324485:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11237.217054] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11237.239889] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [11239.742828] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11239.972827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11245.415861] LustreError: 324485:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11245.425309] LustreError: 324485:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11245.758772] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0255052d800.ost_server_uuid 50 [11247.481708] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0255052d800.ost_server_uuid in FULL state after 0 sec [11253.684334] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0255052d800.ost_server_uuid 50 [11255.401688] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0255052d800.ost_server_uuid in FULL state after 0 sec [11257.802081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11273.187366] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11273.191303] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11273.205537] LustreError: Skipped 1 previous similar message [11273.206800] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11273.206805] Lustre: Skipped 2 previous similar messages [11273.243850] Lustre: Skipped 1 previous similar message [11278.165339] Lustre: server umount lustre-OST0001 complete [11280.360481] LustreError: 323106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11280.374339] LustreError: 323106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11285.476564] LustreError: 323106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11285.509452] LustreError: 323106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11287.531808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11287.558467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11287.593787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11287.605827] Lustre: Skipped 4 previous similar messages [11293.014226] Lustre: server umount lustre-OST0000 complete [11298.784274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11298.794738] LustreError: Skipped 1 previous similar message [11298.798606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11298.807523] Lustre: Skipped 1 previous similar message [11303.085692] Lustre: server umount lustre-MDT0000 complete [11305.956071] LustreError: 323125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.974860] LustreError: 323125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11307.000234] LustreError: 324119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986083 with bad export cookie 16843663988778406621 [11307.014524] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11307.360238] Lustre: server umount lustre-MDT0001 complete [11317.762579] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [11320.608577] Key type lgssc unregistered [11320.936319] LNet: 326826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11320.941451] LNetError: 326826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11320.967394] LNet: Removed LNI 192.168.203.136@tcp [11321.787182] Key type .llcrypt unregistered [11321.792408] Key type ._llcrypt unregistered [11346.836574] Key type ._llcrypt registered [11346.840049] Key type .llcrypt registered [11346.959517] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [11348.112259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11348.136658] alg: No test for adler32 (adler32-zlib) [11349.199861] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11349.438175] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [11351.095493] Key type lgssc registered [11352.349765] Lustre: Echo OBD driver; http://www.lustre.org/ [11360.033487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11360.054272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11361.509554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11365.808457] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11372.002461] LustreError: 328248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.017554] LustreError: 328248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11373.106418] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11377.120335] LustreError: 328249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.068461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.508578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11384.913353] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11392.988497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11393.338053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11399.744662] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11405.607920] LustreError: 329928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.645598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11407.063172] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11411.963373] LustreError: 330548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11411.985993] LustreError: 330548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11414.174529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11414.447675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11419.696203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11421.379317] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11430.144839] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11437.748211] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11438.162790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11542.496682] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11542.511821] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11542.516673] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11542.525934] Lustre: Skipped 1 previous similar message [11542.545379] Lustre: Skipped 1 previous similar message [11547.625167] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11547.633029] Lustre: Skipped 1 previous similar message [11552.746912] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11556.319433] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11556.514274] Lustre: server umount lustre-OST0001 complete [11557.870338] LustreError: 329926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.916179] LustreError: 329926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11562.978976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11563.005790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11563.013015] Lustre: Skipped 1 previous similar message [11563.028190] Lustre: Skipped 2 previous similar messages [11568.108944] LustreError: 329928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11568.109507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11568.131436] LustreError: 329928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11568.153289] Lustre: Skipped 1 previous similar message [11577.311130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11577.494430] Lustre: server umount lustre-OST0000 complete [11583.460561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11583.479812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11583.490035] Lustre: Skipped 2 previous similar messages [11587.679289] Lustre: server umount lustre-MDT0000 complete [11588.576940] LustreError: 331931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.595526] LustreError: 331931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11591.520215] LustreError: 328231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986368 with bad export cookie 5689593958377025101 [11591.522018] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11591.543816] LustreError: 328231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11591.935934] Lustre: server umount lustre-MDT0001 complete [11603.350606] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [11606.211955] Key type lgssc unregistered [11606.510683] LNet: 333196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11606.520151] LNetError: 333196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11606.545937] LNet: Removed LNI 192.168.203.136@tcp [11607.271584] Key type .llcrypt unregistered [11607.276285] Key type ._llcrypt unregistered [11650.507982] Key type ._llcrypt registered [11650.510560] Key type .llcrypt registered [11650.689112] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [11651.596255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11651.729160] alg: No test for adler32 (adler32-zlib) [11652.755917] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [11652.964989] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [11654.671177] Key type lgssc registered [11655.677453] Lustre: Echo OBD driver; http://www.lustre.org/ [11665.490961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11665.514691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11667.091423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11672.381307] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11676.642877] LustreError: 335487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.675147] LustreError: 335487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11681.766169] LustreError: 335488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11682.138340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11682.741919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11687.952366] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11690.930543] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11697.982700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11698.463215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11705.493491] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11707.707555] LustreError: 336978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11711.797639] LustreError: 336978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11711.824226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11711.834414] LustreError: 336978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11716.067251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11716.375715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11721.747125] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11723.113894] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11732.477056] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11743.344577] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11757.538364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.548564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11757.572234] Lustre: Skipped 3 previous similar messages [11757.577817] Lustre: Skipped 2 previous similar messages [11762.659719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11762.670178] Lustre: Skipped 4 previous similar messages [11763.209579] Lustre: server umount lustre-MDT0000 complete [11767.054471] LustreError: 335466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986544 with bad export cookie 9099126478416018348 [11767.071997] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11767.078122] LustreError: 335466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11767.411552] Lustre: server umount lustre-MDT0001 complete [11781.573744] Lustre: server umount lustre-OST0000 complete [11783.145960] Lustre: 334751:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786986544/real 1786986544] req@ffff964c0237ad80 x1873791081841280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786986560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11783.170864] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.026669] Lustre: server umount lustre-OST0001 complete [11803.613315] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [11813.906883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11814.299804] LustreError: 340752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11814.393114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11818.722873] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11824.610426] LustreError: 340753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11824.630215] LustreError: 340753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11827.462111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11827.985402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11833.671181] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11836.959138] Lustre: 341894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11843.306238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11849.195108] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11852.918056] LustreError: 342247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.950744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11857.228407] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11857.234980] Lustre: Skipped 1 previous similar message [11859.310328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11859.320332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11862.252242] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [11869.396776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11872.869799] Lustre: 343764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11878.881249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11878.893349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11878.911480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11883.214928] Lustre: server umount lustre-MDT0000 complete [11885.043296] LustreError: 343071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11885.080100] LustreError: 343071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11886.659884] LustreError: 343767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986663 with bad export cookie 9099126478416020049 [11886.676128] LustreError: 343767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11886.678475] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11887.161093] Lustre: server umount lustre-MDT0001 complete [11902.125978] Lustre: server umount lustre-OST0000 complete [11916.021664] Lustre: server umount lustre-OST0001 complete [11919.787075] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11921.602280] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11923.424206] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 13:11:39 (1786986699) [11949.013137] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [11960.634889] loop0: detected capacity change from 0 to 1099511627776 [11961.504726] SGI XFS with ACLs, security attributes, quota, no debug enabled [11961.522350] *** VALIDATE xfs *** [11961.531068] XFS (loop0): Mounting V5 Filesystem [11961.562383] XFS (loop0): Ending clean mount [11962.844442] loop1: detected capacity change from 0 to 18691697672192 [12070.989791] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [12081.160360] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [12115.192625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12115.440297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12115.463621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12115.533765] Lustre: lustre-MDT0000: new disk, initializing [12115.598535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12115.626748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12119.601592] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12130.306467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12130.435690] Lustre: 348906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12130.515858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12130.522548] Lustre: Skipped 1 previous similar message [12130.652389] Lustre: lustre-MDT0001: new disk, initializing [12130.867124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12130.887480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12135.194281] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12143.796685] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12149.347650] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12156.514389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12156.738895] Lustre: lustre-OST0000: new disk, initializing [12156.742522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12156.747747] Lustre: 350874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12158.332496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12158.354753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12163.266291] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12163.579109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12172.409110] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12177.928917] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12178.273544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12183.468968] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12183.654434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12189.155484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12189.163495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12189.165373] LustreError: Skipped 1 previous similar message [12189.166647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12189.166652] Lustre: Skipped 4 previous similar messages [12189.200122] Lustre: Skipped 4 previous similar messages [12192.115623] Lustre: server umount lustre-OST0000 complete [12197.345126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12197.350451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12197.365314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12197.370056] Lustre: Skipped 1 previous similar message [12202.743761] Lustre: server umount lustre-MDT0000 complete [12204.514525] LustreError: 349763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12204.548847] LustreError: 349763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12206.522059] LustreError: 350873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786986983 with bad export cookie 9099126478416021603 [12206.531496] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12206.535986] LustreError: 350873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12206.911200] Lustre: server umount lustre-MDT0001 complete [12215.293869] XFS (loop0): Unmounting Filesystem [12219.571082] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:16:36 (1786986996) [12226.709946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12227.193921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12227.202488] Lustre: Skipped 2 previous similar messages [12231.959837] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12241.171580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12246.406782] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12254.701338] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12261.251276] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12268.298883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12275.654111] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12283.355382] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12296.673725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12296.684798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.689077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12296.704689] Lustre: Skipped 2 previous similar messages [12296.708684] Lustre: Skipped 2 previous similar messages [12302.397215] Lustre: server umount lustre-OST0000 complete [12308.450313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12313.917900] Lustre: server umount lustre-MDT0000 complete [12318.335705] LustreError: 353412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987095 with bad export cookie 9099126478416022485 [12318.342983] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12318.358084] LustreError: 353412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12318.918854] Lustre: server umount lustre-MDT0001 complete [12332.058754] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [12335.688522] Key type lgssc unregistered [12336.108696] LNet: 357070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12336.124875] LNetError: 357070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12336.141124] LNet: Removed LNI 192.168.203.136@tcp [12337.301170] Key type .llcrypt unregistered [12337.306421] Key type ._llcrypt unregistered [12365.355560] Key type ._llcrypt registered [12365.357429] Key type .llcrypt registered [12365.473668] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [12366.431720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12366.495424] alg: No test for adler32 (adler32-zlib) [12367.682247] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [12367.892249] LNet: Added LNI 192.168.203.136@tcp [8/256/0/180] [12369.623611] Key type lgssc registered [12370.865327] Lustre: Echo OBD driver; http://www.lustre.org/ [12379.748214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12379.768929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12381.200732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12385.981956] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12391.398217] LustreError: 358640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12391.408827] LustreError: 358640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12394.487364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12395.068788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12399.907404] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12408.787456] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12415.866487] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12423.878462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12424.288687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12430.601996] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12438.949916] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12445.158348] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12445.171941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12445.179857] Lustre: Skipped 1 previous similar message [12445.966675] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.36@tcp (stopping) [12447.417116] Lustre: server umount lustre-MDT0001 complete [12450.276495] LustreError: 358640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.294449] LustreError: 358640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12455.402020] LustreError: 358637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12455.426696] LustreError: 358637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12459.372525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12459.524417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12459.529643] Lustre: Found index 1 for lustre-MDT0001, updating log [12459.568699] Lustre: 358630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12459.977396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12465.161731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12465.966172] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12468.718949] LustreError: 362198:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12468.723787] LustreError: 362198:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12468.729972] LustreError: 358632:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12468.758107] LustreError: 362201:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.136@tcp: cfg command failed: rc = -17 [12468.765980] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12468.765980] [12476.697233] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.36@tcp (stopping) [12480.483795] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12480.490352] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12480.496249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12480.504967] Lustre: Skipped 2 previous similar messages [12482.584829] Lustre: server umount lustre-MDT0001 complete [12485.619777] LustreError: 358640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12485.630254] LustreError: 358640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12490.728201] LustreError: 358641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.739789] LustreError: 358641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12492.568387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.683920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12492.692521] Lustre: Found index 1 for lustre-MDT0001, updating log [12492.746162] Lustre: 358628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12493.094768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12496.884842] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12498.420681] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12498.948154] LustreError: 363279:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12498.970784] LustreError: 363279:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12498.999851] LustreError: 358632:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12499.017214] LustreError: 358632:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12499.093582] LustreError: 363285:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12499.108742] LustreError: 363285:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12499.113994] LustreError: 363285:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.136@tcp: cfg command failed: rc = -17 [12499.120331] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.136@tcp [12499.120331] [12802.343905] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.36@tcp (stopping) [12802.355218] Lustre: Skipped 2 previous similar messages [12805.606191] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12805.613903] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12805.622055] Lustre: Skipped 1 previous similar message [12810.723636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12810.732727] Lustre: Skipped 4 previous similar messages [12816.351136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12816.692957] Lustre: server umount lustre-MDT0001 complete [12817.656033] LustreError: 365682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.685909] LustreError: 365682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12827.594126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12827.737202] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12827.746485] Lustre: Found index 1 for lustre-MDT0001, updating log [12827.836581] Lustre: 358629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12828.190596] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12833.266322] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12833.465237] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [12834.811090] LustreError: 371749:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12834.817855] LustreError: 371749:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12834.822726] LustreError: 358632:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12834.830751] LustreError: 358632:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12834.932065] LustreError: 371758:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12834.954645] LustreError: 371758:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.136@tcp: cfg command failed: rc = -114 [12834.971727] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12834.971727] [13152.024477] LustreError: 358622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.36@tcp arrived at 1786987928 with bad export cookie 8172559185488325740 [13152.025122] LustreError: 358620:0:(ldlm_lock.c:2769:ldlm_lock_dump_handle()) ### ### ns: MGS lock: 000000006ac0f986/0x716ac34774e3312e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x40000000000000 nid: 192.168.203.36@tcp remote: 0xfe110789fad18569 expref: 5 pid: 358629 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [13152.073608] LustreError: 358622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13155.817114] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13155.820285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13155.833732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13155.833741] Lustre: Skipped 3 previous similar messages [13155.837428] Lustre: Skipped 1 previous similar message [13160.936696] LustreError: 365682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13160.943088] LustreError: 365682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [13161.028831] Lustre: server umount lustre-MDT0000 complete [13165.955406] LustreError: 358622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786987942 with bad export cookie 8172559185488325075 [13165.958673] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13165.979125] LustreError: 358622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13166.061783] LustreError: 357950:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [13166.070552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13166.086851] Lustre: Skipped 1 previous similar message [13180.383356] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13181.017570] Lustre: server umount lustre-MDT0001 complete [13186.158577] Lustre: server umount lustre-OST0000 complete [13198.383749] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [13209.871264] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [13248.575927] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:33:45 (1786988025) [13255.406489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13255.678202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13255.718445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13255.797701] Lustre: lustre-MDT0000: new disk, initializing [13255.873462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13255.905796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13260.134916] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [13271.995697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13272.161243] Lustre: 382570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13272.216144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13272.219705] Lustre: Skipped 1 previous similar message [13272.295864] Lustre: lustre-MDT0001: new disk, initializing [13272.408259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13272.493517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13272.523582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13276.955915] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [13286.427604] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13291.794416] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13299.476113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13300.096138] Lustre: lustre-OST0000: new disk, initializing [13300.105714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13300.114916] Lustre: 384539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13300.353827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13309.042613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13309.065180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13309.263726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13310.630783] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_default_debug -1 all [13322.429974] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13328.867048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13328.882962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13328.906682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13328.924223] Lustre: Skipped 5 previous similar messages [13334.827521] Lustre: server umount lustre-MDT0000 complete [13335.018940] LustreError: 382576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.063313] LustreError: 382576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13338.399898] LustreError: 382561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786988115 with bad export cookie 8172559185488327028 [13338.409880] LustreError: MGC192.168.203.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13338.411407] LustreError: 382561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13338.424749] LustreError: 383376:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13338.723422] Lustre: server umount lustre-MDT0001 complete [13352.493920] Lustre: server umount lustre-OST0000 complete [13382.923320] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing set_hostid [13394.227614] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing load_modules_local [13427.103735] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13166 sec ========= 13:36:43 (1786988203) [13429.010123] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:36:45 (1786988205) === [13432.664293] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:36:49 (1786988209) === [13474.787170] Lustre: DEBUG MARKER: oleg336-server.virtnet: executing unload_modules_local [13477.468957] Key type lgssc unregistered [13477.759522] LNet: 390912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13477.769408] LNetError: 390912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13477.786892] LNet: Removed LNI 192.168.203.136@tcp [13478.599109] Key type .llcrypt unregistered [13478.601347] Key type ._llcrypt unregistered