[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499775310 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003287] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011113] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022247] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027303] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030087] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031131] *** VALIDATE tmpfs *** [ 0.032462] *** VALIDATE proc *** [ 0.034068] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037059] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040034] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043730] debug: unmapping init [mem 0xffffffff86259000-0xffffffff86260fff] [ 0.045165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046681] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054296] rcu: Hierarchical SRCU implementation. [ 0.056445] smp: Bringing up secondary CPUs ... [ 0.057590] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.062027] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137181] node 0 deferred pages initialised in 69ms [ 0.141110] devtmpfs: initialized [ 0.142262] x86/mm: Memory block size: 128MB [ 0.146469] gcov: version magic: 0x41383552 [ 0.150454] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157328] pinctrl core: initialized pinctrl subsystem [ 0.159202] [ 0.159751] ************************************************************* [ 0.162015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165016] ** ** [ 0.167014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170017] ** ** [ 0.172016] ** This means that this kernel is built to expose internal ** [ 0.175019] ** IOMMU data structures, which may compromise security on ** [ 0.178020] ** your system. ** [ 0.180018] ** ** [ 0.183021] ** If you see this message and you are not debugging the ** [ 0.185012] ** kernel, report this immediately to your vendor! ** [ 0.188012] ** ** [ 0.190019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193012] ************************************************************* [ 0.196174] NET: Registered protocol family 16 [ 0.198847] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209436] cpuidle: using governor menu [ 0.212186] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214383] PCI: Using configuration type 1 for base access [ 0.216103] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231125] cryptd: max_cpu_qlen set to 1000 [ 0.234308] ACPI: Added _OSI(Module Device) [ 0.236018] ACPI: Added _OSI(Processor Device) [ 0.238015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244407] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251565] ACPI: Interpreter enabled [ 0.253049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254009] ACPI: Using IOAPIC for interrupt routing [ 0.255094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260383] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270201] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280331] acpiphp: Slot [2] registered [ 0.282121] acpiphp: Slot [5] registered [ 0.284149] acpiphp: Slot [6] registered [ 0.286134] acpiphp: Slot [7] registered [ 0.287143] acpiphp: Slot [8] registered [ 0.288197] acpiphp: Slot [9] registered [ 0.290099] acpiphp: Slot [10] registered [ 0.291106] acpiphp: Slot [3] registered [ 0.292090] acpiphp: Slot [4] registered [ 0.294105] acpiphp: Slot [11] registered [ 0.295153] acpiphp: Slot [12] registered [ 0.297123] acpiphp: Slot [13] registered [ 0.298118] acpiphp: Slot [14] registered [ 0.299139] acpiphp: Slot [15] registered [ 0.300055] acpiphp: Slot [16] registered [ 0.301113] acpiphp: Slot [17] registered [ 0.302099] acpiphp: Slot [18] registered [ 0.304128] acpiphp: Slot [19] registered [ 0.306109] acpiphp: Slot [20] registered [ 0.308095] acpiphp: Slot [21] registered [ 0.309081] acpiphp: Slot [22] registered [ 0.310103] acpiphp: Slot [23] registered [ 0.312105] acpiphp: Slot [24] registered [ 0.313091] acpiphp: Slot [25] registered [ 0.315092] acpiphp: Slot [26] registered [ 0.316154] acpiphp: Slot [27] registered [ 0.317107] acpiphp: Slot [28] registered [ 0.319108] acpiphp: Slot [29] registered [ 0.321360] acpiphp: Slot [30] registered [ 0.322108] acpiphp: Slot [31] registered [ 0.323067] PCI host bridge to bus 0000:00 [ 0.325016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341792] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345381] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.360051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.367017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369481] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371627] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378594] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.410870] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.453166] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.511000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.533016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.565232] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.632358] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.641016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.647015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.667014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682116] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.689026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.697015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.727715] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.730345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.732322] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.734360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.736287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739129] iommu: Default domain type: Passthrough [ 0.741412] SCSI subsystem initialized [ 0.742206] ACPI: bus type USB registered [ 0.744108] usbcore: registered new interface driver usbfs [ 0.746105] usbcore: registered new interface driver hub [ 0.747110] usbcore: registered new device driver usb [ 0.749163] pps_core: LinuxPPS API ver. 1 registered [ 0.750008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.753061] PTP clock support registered [ 0.755048] EDAC MC: Ver: 3.0.0 [ 0.756543] PCI: Using ACPI for IRQ routing [ 0.758795] NetLabel: Initializing [ 0.760010] NetLabel: domain hash size = 128 [ 0.761009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763091] NetLabel: unlabeled traffic allowed by default [ 0.766078] vgaarb: loaded [ 0.767325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.769017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.774455] clocksource: Switched to clocksource kvm-clock [ 0.883832] VFS: Disk quotas dquot_6.6.0 [ 0.885185] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886979] *** VALIDATE ramfs *** [ 0.888259] *** VALIDATE hugetlbfs *** [ 0.889989] pnp: PnP ACPI init [ 0.892110] pnp: PnP ACPI: found 6 devices [ 0.921436] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.924879] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.927130] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.929287] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.931879] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.934894] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.937784] NET: Registered protocol family 2 [ 0.939744] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.944740] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.948567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.954176] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958219] TCP: Hash tables configured (established 65536 bind 65536) [ 0.960756] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964294] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967302] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.970755] NET: Registered protocol family 1 [ 0.976159] RPC: Registered named UNIX socket transport module. [ 0.979013] RPC: Registered udp transport module. [ 0.980672] RPC: Registered tcp transport module. [ 0.982675] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985800] NET: Registered protocol family 44 [ 0.988182] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990908] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.994455] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.998914] PCI: CLS 0 bytes, default 64 [ 1.001396] Unpacking initramfs... [ 2.406425] debug: unmapping init [mem 0xffff989a3cc54000-0xffff989a3ffbffff] [ 2.410521] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.412777] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.415570] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.958783] Initialise system trusted keyrings [ 2.960376] Key type blacklist registered [ 2.962658] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.970406] zbud: loaded [ 2.973279] *** VALIDATE nfs *** [ 2.974234] *** VALIDATE nfs4 *** [ 2.976279] pstore: using deflate compression [ 2.983151] Platform Keyring initialized [ 3.081897] NET: Registered protocol family 38 [ 3.083602] Key type asymmetric registered [ 3.084860] Asymmetric key parser 'x509' registered [ 3.086533] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.089263] io scheduler mq-deadline registered [ 3.090987] io scheduler kyber registered [ 3.092583] io scheduler bfq registered [ 3.094390] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.096930] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.099632] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.102169] ACPI: Power Button [PWRF] [ 3.107475] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.114478] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.129116] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.135773] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.154788] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.186124] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.216090] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.221519] Non-volatile memory driver v1.3 [ 3.223951] Linux agpgart interface v0.103 [ 3.253552] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.257607] vda: detected capacity change from 0 to 76775424 [ 3.271080] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.274480] vdb: detected capacity change from 0 to 1073741824 [ 3.290892] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.293828] vdc: detected capacity change from 0 to 2621440000 [ 3.310229] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.313040] vdd: detected capacity change from 0 to 2621440000 [ 3.329261] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.332141] vde: detected capacity change from 0 to 4294967296 [ 3.353069] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.356737] vdf: detected capacity change from 0 to 4294967296 [ 3.364192] libphy: Fixed MDIO Bus: probed [ 3.369962] usbcore: registered new interface driver usbserial_generic [ 3.372429] usbserial: USB Serial support registered for generic [ 3.374495] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.378916] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.381105] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.383684] mousedev: PS/2 mouse device common for all mice [ 3.388287] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.388623] rtc_cmos 00:05: RTC can wake from S4 [ 3.397823] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.398098] rtc_cmos 00:05: registered as rtc0 [ 3.405937] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.410109] intel_pstate: CPU model not supported [ 3.411682] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.417953] hid: raw HID events driver (C) Jiri Kosina [ 3.420134] usbcore: registered new interface driver usbhid [ 3.422252] usbhid: USB HID core driver [ 3.424791] drop_monitor: Initializing network drop monitor service [ 3.426684] Initializing XFRM netlink socket [ 3.428920] NET: Registered protocol family 10 [ 3.432253] Segment Routing with IPv6 [ 3.433607] NET: Registered protocol family 17 [ 3.435969] mpls_gso: MPLS GSO support [ 3.441582] RAS: Correctable Errors collector initialized. [ 3.443624] AVX version of gcm_enc/dec engaged. [ 3.444908] AES CTR mode by8 optimization enabled [ 3.518785] sched_clock: Marking stable (3518701244, 0)->(4468698626, -949997382) [ 3.521850] registered taskstats version 1 [ 3.523783] Loading compiled-in X.509 certificates [ 3.525945] zswap: loaded using pool lzo/zbud [ 3.550236] Key type big_key registered [ 3.565863] Key type encrypted registered [ 3.567407] ima: No TPM chip found, activating TPM-bypass! [ 3.569695] ima: Allocated hash algorithm: sha1 [ 3.571360] ima: No architecture policies found [ 3.572821] evm: Initialising EVM extended attributes: [ 3.574724] evm: security.selinux [ 3.575954] evm: security.ima [ 3.577179] evm: security.capability [ 3.578505] evm: HMAC attrs: 0x1 [ 3.580773] rtc_cmos 00:05: setting system clock to 2026-09-09 07:51:25 UTC (1788940285) [ 3.586970] debug: unmapping init [mem 0xffffffff87203000-0xffffffff873fffff] [ 3.590162] debug: unmapping init [mem 0xffffffff85f82000-0xffffffff86258fff] [ 3.601171] Write protecting the kernel read-only data: 28672k [ 3.604512] debug: unmapping init [mem 0xffffffff84603000-0xffffffff847fffff] [ 3.606346] debug: unmapping init [mem 0xffffffff84f14000-0xffffffff84ffffff] [ 3.643150] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.651984] systemd[1]: Detected virtualization kvm. [ 3.653966] systemd[1]: Detected architecture x86-64. [ 3.655906] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.682144] systemd[1]: No hostname configured. [ 3.683866] systemd[1]: Set hostname to . [ 3.685820] random: systemd: uninitialized urandom read (16 bytes read) [ 3.688191] systemd[1]: Initializing machine ID from random generator. [ 3.730681] random: ln: uninitialized urandom read (6 bytes read) [ 3.816934] random: systemd: uninitialized urandom read (16 bytes read) [ 3.820190] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.823541] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.827831] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.448975] device-mapper: uevent: version 1.0.3 [ 4.451366] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.119984] random: fast init done [ 5.152709] virtio_net virtio0 ens2: renamed from eth0 [ 5.157204] scsi host0: ata_piix [ 5.161129] scsi host1: ata_piix [ 5.162482] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.165577] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.879677] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.085433] random: crng init done [ 10.087326] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.460605] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.628868] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.906552] SELinux: Disabled at runtime. [ 11.968691] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.978245] systemd[1]: Detected virtualization kvm. [ 11.980449] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.488780] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.493068] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.500919] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.504044] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.506542] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.513015] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.518449] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ 12.693877] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.109885] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.499126] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.510147] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.659457] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.676459] EDAC sbridge: Ver: 1.1.2 [ 15.664948] Key type dns_resolver registered [ 15.994440] NFS: Registering the id_resolver key type [ 15.998722] Key type id_resolver registered [ 16.001565] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg421-server login: [ 46.099263] libcfs: loading out-of-tree module taints kernel. [ 46.126107] Key type ._llcrypt registered [ 46.127957] Key type .llcrypt registered [ 46.238202] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 70.584230] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 72.587226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.600323] alg: No test for adler32 (adler32-zlib) [ 74.455783] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 75.099026] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 76.944200] Key type lgssc registered [ 78.667794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.968914] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 98.576053] hrtimer: interrupt took 2928125 ns [ 145.308600] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 159.120891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.155839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.412255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.479617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.688332] Lustre: lustre-MDT0000: new disk, initializing [ 160.835774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.869381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.982796] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.047730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.140274] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.163822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.167890] Lustre: Skipped 1 previous similar message [ 182.254966] Lustre: lustre-MDT0001: new disk, initializing [ 182.329401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.361469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.399252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.617449] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.171320] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.499859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.741494] Lustre: lustre-OST0000: new disk, initializing [ 204.744860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.749141] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.805917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.991503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.009233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.165382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.216886] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.252766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.369612] Lustre: lustre-OST0001: new disk, initializing [ 227.375948] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 227.383640] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.429615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.685541] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.057759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.064073] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.132435] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.948059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.471496] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.933415] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing check_logdir /tmp/testlogs/ [ 267.049767] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing yml_node [ 273.604078] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 276.878978] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 279.863690] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 282.016896] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 03:56:00 EDT 2026 [ 300.781057] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 302.803255] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.481049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 320.497284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.515468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.018473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.024395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.053229] Lustre: Skipped 3 previous similar messages [ 325.842399] Lustre: server umount lustre-MDT0000 complete [ 329.635237] LustreError: 10338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940611 with bad export cookie 1781833973204945988 [ 329.641945] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 329.676533] LustreError: 10338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 332.260406] LustreError: 9526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.261648] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.265681] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.291875] LustreError: 9526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 332.370982] Lustre: Skipped 3 previous similar messages [ 334.307185] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.310262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.340579] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 334.388562] Lustre: Skipped 2 previous similar messages [ 336.081422] Lustre: server umount lustre-MDT0001 complete [ 341.284573] Lustre: server umount lustre-OST0000 complete [ 345.778963] Lustre: server umount lustre-OST0001 complete [ 352.950658] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 369.241476] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 414.324781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.736521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.816942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.933686] Lustre: lustre-MDT0000: new disk, initializing [ 415.113796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.142858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.424833] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 433.167676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.331146] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.458871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.467545] Lustre: Skipped 1 previous similar message [ 433.622794] Lustre: lustre-MDT0001: new disk, initializing [ 433.854712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.932350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.953659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.399594] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 450.065722] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 456.759269] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.928161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.322182] Lustre: lustre-OST0000: new disk, initializing [ 465.326966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.333514] Lustre: 17068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.428343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.114630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.124546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.246701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.565984] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 482.584800] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.307974] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.570576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 494.013854] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 494.245379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.149050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.150878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.199128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.771758] Lustre: server umount lustre-OST0000 complete [ 508.389167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.405908] Lustre: Skipped 1 previous similar message [ 508.412656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.421206] Lustre: Skipped 2 previous similar messages [ 513.046218] Lustre: server umount lustre-MDT0000 complete [ 513.512319] LustreError: 15112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.533169] LustreError: 15112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 517.167317] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940799 with bad export cookie 1781833973204947521 [ 517.169241] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.181640] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 517.688509] Lustre: server umount lustre-MDT0001 complete [ 523.051503] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 04:00:02 (1788940802) [ 530.333778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.953796] LustreError: 19334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.106894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.757887] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 536.564847] LustreError: 19335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.586220] LustreError: 19335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 545.763150] LustreError: 19335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.791599] LustreError: 19335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 546.640782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.687370] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 563.926329] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.247918] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.788643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 588.040094] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 596.336222] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.960395] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 616.418045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.435752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.467708] Lustre: Skipped 2 previous similar messages [ 616.508145] Lustre: Skipped 2 previous similar messages [ 620.088699] Lustre: server umount lustre-MDT0000 complete [ 621.540425] LustreError: 20062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.554401] LustreError: 20062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 624.714832] LustreError: 19315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940906 with bad export cookie 1781833973204948403 [ 624.722326] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.731492] LustreError: 19315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 625.365138] Lustre: server umount lustre-MDT0001 complete [ 640.798886] Lustre: server umount lustre-OST0000 complete [ 664.440660] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 677.086060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.697491] LustreError: 23731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.795746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.802329] Lustre: Skipped 2 previous similar messages [ 684.877591] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 694.977206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.437969] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 702.641426] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 710.973883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.867807] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 727.288889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 727.466127] Lustre: lustre-OST0001: new disk, initializing [ 727.468614] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 727.472145] Lustre: 26050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 729.114582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 729.120694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 729.247350] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 734.701210] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 747.847312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 752.501878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 770.023722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.029668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 770.053724] Lustre: Skipped 2 previous similar messages [ 770.063532] Lustre: Skipped 1 previous similar message [ 776.063640] Lustre: server umount lustre-MDT0000 complete [ 780.262995] LustreError: 23732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 780.292665] LustreError: 23732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 783.172394] LustreError: 27020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941065 with bad export cookie 1781833973204949250 [ 783.179246] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 783.197928] LustreError: 27020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 783.623498] Lustre: server umount lustre-MDT0001 complete [ 798.283504] Lustre: server umount lustre-OST0000 complete [ 801.121405] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788941067/real 1788941067] req@ffff9899894cad80 x1875839924039296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788941083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 803.469924] Lustre: server umount lustre-OST0001 complete [ 823.014170] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 834.428504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.139332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.151925] Lustre: Skipped 3 previous similar messages [ 840.572941] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 850.445022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.252524] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 859.264138] Lustre: 30223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 866.856626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.068253] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 885.556427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 892.160981] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 900.891468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 905.388350] Lustre: 32104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 916.967712] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.969640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.998798] Lustre: Skipped 5 previous similar messages [ 917.025155] Lustre: Skipped 9 previous similar messages [ 920.117246] Lustre: server umount lustre-MDT0000 complete [ 922.089420] LustreError: 29073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.125177] LustreError: 29073:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 923.917191] LustreError: 29055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941205 with bad export cookie 1781833973204950538 [ 923.925687] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 923.930479] LustreError: 29055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 924.237525] Lustre: server umount lustre-MDT0001 complete [ 938.683018] Lustre: server umount lustre-OST0000 complete [ 953.486855] Lustre: server umount lustre-OST0001 complete [ 963.573294] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 04:07:22 (1788941242) [ 992.379852] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1004.945848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.598506] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1021.772218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.788356] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1030.771043] Lustre: 36062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1038.378565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.400583] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1055.335256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1062.453257] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1070.443928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1073.840679] Lustre: 37940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1078.768032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.788217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.803798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.660157] Lustre: server umount lustre-MDT0000 complete [ 1089.231625] LustreError: 34891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941371 with bad export cookie 1781833973204951651 [ 1089.233591] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.242046] LustreError: 34891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1089.686200] Lustre: server umount lustre-MDT0001 complete [ 1104.833395] Lustre: server umount lustre-OST0000 complete [ 1107.937716] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788941373/real 1788941373] req@ffff989a9f59ce00 x1875839924144256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788941389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1108.940236] Lustre: server umount lustre-OST0001 complete [ 1119.250915] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 04:09:58 (1788941398) [ 1135.512147] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1145.240411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.791992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.796130] Lustre: Skipped 7 previous similar messages [ 1150.053307] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1159.225549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.372707] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1166.887763] Lustre: 41227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1175.380069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1182.721963] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1184.173619] LustreError: 41582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.213149] LustreError: 41582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 1191.550707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1198.258552] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1205.972527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1209.813771] Lustre: 43105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1210.772655] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1210.786376] Lustre: Skipped 1 previous similar message [ 1215.971140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1215.981202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.999437] Lustre: Skipped 4 previous similar messages [ 1219.950626] Lustre: server umount lustre-MDT0000 complete [ 1223.478523] LustreError: 43108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941505 with bad export cookie 1781833973204952757 [ 1223.487380] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.487957] LustreError: 43108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1230.128325] Lustre: server umount lustre-MDT0001 complete [ 1238.177338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.300970] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1260.031423] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1262.775422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.453278] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1268.740359] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1268.749218] Lustre: Skipped 1 previous similar message [ 1277.570478] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.156804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1285.453333] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1287.178443] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1293.798799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1299.071560] Lustre: server umount lustre-MDT0000 complete [ 1303.233756] LustreError: 41229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941585 with bad export cookie 1781833973204953856 [ 1303.239335] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.247342] LustreError: 41229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1303.697112] Lustre: server umount lustre-MDT0001 complete [ 1318.055836] Lustre: server umount lustre-OST0000 complete [ 1320.929158] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788941586/real 1788941586] req@ffff989ab1b94000 x1875839924233088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788941602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1322.266411] Lustre: server umount lustre-OST0001 complete [ 1332.163515] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 04:13:31 (1788941611) [ 1348.348662] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1359.023035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.568793] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1372.563491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.567322] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1380.843746] Lustre: 48868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1387.964548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.513112] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1403.603484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1410.599851] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1418.562961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1424.047604] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1439.713178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.725833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.736731] Lustre: Skipped 20 previous similar messages [ 1443.533708] Lustre: server umount lustre-MDT0000 complete [ 1447.461320] LustreError: 49252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941729 with bad export cookie 1781833973204954675 [ 1447.465476] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1447.468222] LustreError: 49252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1447.876655] Lustre: server umount lustre-MDT0001 complete [ 1462.150473] Lustre: server umount lustre-OST0000 complete [ 1476.258962] Lustre: server umount lustre-OST0001 complete [ 1485.602415] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 04:16:04 (1788941764) [ 1488.253128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1489.973398] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 04:16:09 (1788941769) [ 1500.885959] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1510.460938] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1534.195663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.519898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1534.602165] Lustre: lustre-MDT0000: new disk, initializing [ 1534.708716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1538.861668] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1548.099512] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1554.524335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.817610] Lustre: lustre-OST0000: new disk, initializing [ 1554.820325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1554.826089] Lustre: Skipped 1 previous similar message [ 1554.829530] Lustre: 54957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1556.840836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1556.855992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1556.897058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1562.038565] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1572.082414] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1574.647274] Lustre: DEBUG MARKER: create test files [ 1608.172087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1608.179247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.204849] Lustre: Skipped 14 previous similar messages [ 1611.405453] Lustre: server umount lustre-OST0000 complete [ 1616.355797] Lustre: server umount lustre-MDT0000 complete [ 1627.647907] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1630.677329] Key type lgssc unregistered [ 1630.990330] LNet: 56587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.999445] LNetError: 56587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.022271] LNet: Removed LNI 192.168.204.121@tcp [ 1631.979953] Key type .llcrypt unregistered [ 1631.987838] Key type ._llcrypt unregistered [ 1661.778340] Key type ._llcrypt registered [ 1661.781136] Key type .llcrypt registered [ 1661.914709] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1663.194486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1663.265735] alg: No test for adler32 (adler32-zlib) [ 1664.511616] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1664.904544] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1666.680294] Key type lgssc registered [ 1667.860549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1675.937170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1675.980141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.542270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.907256] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1690.215985] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1698.348881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1698.695203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.782771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1702.988537] Lustre: 59319:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1703.000047] Lustre: 59319:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1703.008903] Lustre: 59319:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1703.029837] Lustre: 59319:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1703.049108] Lustre: 59319:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1703.061023] Lustre: 59319:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1706.291225] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1714.139523] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1720.073722] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1725.819531] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1738.721180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1738.735116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1738.747523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.092729] Lustre: server umount lustre-OST0000 complete [ 1745.767842] Lustre: server umount lustre-MDT0000 complete [ 1759.307234] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1762.524544] Key type lgssc unregistered [ 1762.881277] LNet: 60928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.896170] LNetError: 60928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.919935] LNet: Removed LNI 192.168.204.121@tcp [ 1764.033444] Key type .llcrypt unregistered [ 1764.037154] Key type ._llcrypt unregistered [ 1795.445534] Key type ._llcrypt registered [ 1795.450667] Key type .llcrypt registered [ 1795.587187] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1797.056916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.146597] alg: No test for adler32 (adler32-zlib) [ 1798.258814] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1798.534442] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1800.192252] Key type lgssc registered [ 1801.561920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.290743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1810.324233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.836953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.998775] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1825.436518] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1833.050808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1833.360621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.467836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1836.592937] Lustre: 63758:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1836.609557] Lustre: 63758:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1836.616050] Lustre: 63758:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1836.627824] Lustre: 63758:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1836.637781] Lustre: 63758:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1836.645153] Lustre: 63758:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1840.341897] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1849.283795] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1855.812399] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1867.234118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.237609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.257241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.306245] Lustre: server umount lustre-OST0000 complete [ 1875.992290] Lustre: server umount lustre-MDT0000 complete [ 1889.766253] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1893.611871] Key type lgssc unregistered [ 1893.842628] LNet: 65314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.855934] LNetError: 65314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.911342] LNet: Removed LNI 192.168.204.121@tcp [ 1895.978412] Key type .llcrypt unregistered [ 1895.988585] Key type ._llcrypt unregistered [ 1917.208900] Key type ._llcrypt registered [ 1917.215439] Key type .llcrypt registered [ 1917.345321] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1934.315811] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1935.681922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.922158] alg: No test for adler32 (adler32-zlib) [ 1937.399293] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 1937.841471] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1939.632346] Key type lgssc registered [ 1941.287600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1981.984239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1982.013443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.376268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1983.427531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1983.558233] Lustre: lustre-MDT0000: new disk, initializing [ 1983.699908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.742279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1988.689540] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2000.597846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.814311] Lustre: 68470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2000.934748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2000.944919] Lustre: Skipped 1 previous similar message [ 2001.105714] Lustre: lustre-MDT0001: new disk, initializing [ 2001.256669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2001.314640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2001.331828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2006.536734] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2016.059488] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.071765] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2029.953610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2030.278974] Lustre: lustre-OST0000: new disk, initializing [ 2030.285873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2030.296830] Lustre: 70438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2030.406551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.602721] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2039.409905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2039.420334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2039.560426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2049.088354] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2054.598545] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2054.796961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2059.986700] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2060.304534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2065.380441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2065.389142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.406036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2065.419177] Lustre: Skipped 1 previous similar message [ 2065.433188] Lustre: Skipped 1 previous similar message [ 2068.768087] Lustre: server umount lustre-OST0000 complete [ 2073.060751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2073.069906] LustreError: Skipped 1 previous similar message [ 2073.077298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.094116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2075.621404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.627960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2075.660914] Lustre: Skipped 1 previous similar message [ 2080.740224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.752151] Lustre: Skipped 1 previous similar message [ 2085.857486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.869663] Lustre: Skipped 1 previous similar message [ 2086.880192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2087.158097] Lustre: server umount lustre-MDT0000 complete [ 2091.186958] LustreError: 68461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942373 with bad export cookie 2591475699510310521 [ 2091.189980] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.193016] LustreError: 68461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2091.566190] Lustre: server umount lustre-MDT0001 complete [ 2100.590590] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 04:26:19 (1788942379) [ 2111.928166] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2151.233681] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2159.561971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2192.372311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.570552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2192.603064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2192.682643] Lustre: lustre-MDT0000: new disk, initializing [ 2192.774511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.801372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2197.661675] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2207.983378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.096225] Lustre: 76460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2208.136248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2208.139519] Lustre: Skipped 1 previous similar message [ 2208.200766] Lustre: lustre-MDT0001: new disk, initializing [ 2208.316921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2208.360249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2208.370944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2213.452351] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2223.727235] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2229.772716] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.163239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2237.479275] Lustre: lustre-OST0000: new disk, initializing [ 2237.485976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2237.497273] Lustre: 78429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2237.607062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.725767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2243.741651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2243.850683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2244.174922] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2253.603905] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2258.778618] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2259.028477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2263.971932] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2264.216439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2269.670514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2269.672583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.677671] LustreError: Skipped 1 previous similar message [ 2269.677739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2269.688891] Lustre: Skipped 1 previous similar message [ 2269.715082] Lustre: Skipped 1 previous similar message [ 2272.374776] Lustre: server umount lustre-OST0000 complete [ 2279.910219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.914369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2279.924464] Lustre: Skipped 1 previous similar message [ 2282.718188] Lustre: server umount lustre-MDT0000 complete [ 2285.028684] LustreError: 76472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.067265] LustreError: 76472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2286.427284] LustreError: 78428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942568 with bad export cookie 2591475699510311403 [ 2286.431965] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.438078] LustreError: 78428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2287.032262] Lustre: server umount lustre-MDT0001 complete [ 2295.729785] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 04:29:34 (1788942574) [ 2303.035549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.669865] LustreError: 80841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.755974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.363647] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2309.103958] LustreError: 80842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.187980] LustreError: 80841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.891822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.026601] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2329.471342] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.478996] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2342.675164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.823300] LustreError: 82664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.115522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.126607] Lustre: Skipped 1 previous similar message [ 2350.153485] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2357.872561] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2361.630532] Lustre: 83514:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting efde7fc5-0f46-4042-b99f-5eb6bb8909d2 at adminstrative request [ 2363.892467] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.911589] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2373.097295] Lustre: MGS: Client efde7fc5-0f46-4042-b99f-5eb6bb8909d2 (at 0@lo) reconnecting [ 2373.107411] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x23f6c56b215e7d5d to 0x23f6c56b215e7ffd [ 2373.126301] Lustre: MGC192.168.204.121@tcp: Connection restored to 0@lo (at 0@lo) [ 2399.115504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.385672] Lustre: lustre-OST0001: new disk, initializing [ 2399.392566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2399.409223] Lustre: 83911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2405.591779] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2406.019409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2406.029120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2406.186077] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2415.365752] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.729690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.740382] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.745903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2421.769736] Lustre: Skipped 1 previous similar message [ 2421.772496] Lustre: Skipped 2 previous similar messages [ 2424.473667] Lustre: server umount lustre-OST0001 complete [ 2426.854282] LustreError: 82666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.910803] LustreError: 82666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2430.434402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.445863] LustreError: Skipped 2 previous similar messages [ 2435.348351] Lustre: server umount lustre-OST0000 complete [ 2440.165532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.194645] Lustre: Skipped 2 previous similar messages [ 2445.304430] Lustre: server umount lustre-MDT0000 complete [ 2447.329080] LustreError: 80842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.337988] LustreError: 80842:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2448.855693] LustreError: 80823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942730 with bad export cookie 2591475699510312957 [ 2448.859540] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.886045] LustreError: 80823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2449.341264] Lustre: server umount lustre-MDT0001 complete [ 2457.020717] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 04:32:15 (1788942735) [ 2459.040474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2460.659236] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 04:32:20 (1788942740) [ 2462.263291] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2464.293531] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 04:32:23 (1788942743) [ 2466.099424] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2467.881582] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 04:32:26 (1788942746) [ 2469.530481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2471.612389] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 04:32:30 (1788942750) [ 2473.814760] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2480.108727] loop: module loaded [ 2480.134726] loop0: detected capacity change from 0 to 4294967296 [ 2480.177020] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2482.063579] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2486.928833] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2491.527479] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2520.544956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.579665] LustreError: 88034:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2520.590478] LustreError: 88034:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2520.599714] LustreError: 88034:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2520.605755] LustreError: 88034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2539.926766] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2550.347437] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2583.102204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.293379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2583.318337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2583.395892] Lustre: lustre-MDT0000: new disk, initializing [ 2583.478242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.484264] Lustre: Skipped 1 previous similar message [ 2583.497438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2587.550227] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2597.403663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.522944] Lustre: 90901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2597.843564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2602.005847] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2610.620493] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2615.878666] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2622.932168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2623.255021] Lustre: 92871:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2630.193084] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2631.283414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2631.424447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2639.604890] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.273304] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2645.521957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2650.658468] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2650.841130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2657.250309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2657.254272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.269847] LustreError: Skipped 2 previous similar messages [ 2657.271253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.285629] Lustre: Skipped 2 previous similar messages [ 2657.311950] Lustre: Skipped 8 previous similar messages [ 2659.244399] Lustre: server umount lustre-OST0000 complete [ 2669.546183] Lustre: server umount lustre-MDT0000 complete [ 2672.620847] LustreError: 90913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.648427] LustreError: 90913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2673.574176] LustreError: 91707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942955 with bad export cookie 2591475699510313573 [ 2673.579310] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2673.591070] LustreError: 91707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2673.855451] Lustre: server umount lustre-MDT0001 complete [ 2682.674634] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 04:36:01 (1788942961) [ 2689.203210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.556136] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2702.748604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.345852] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2714.558263] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2719.657623] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2725.937993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.288287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.295039] Lustre: Skipped 4 previous similar messages [ 2732.341739] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2740.515070] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2745.529971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2745.770685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2750.634043] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2750.912199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2757.882241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.120198] Lustre: lustre-OST0001: new disk, initializing [ 2758.122909] Lustre: Skipped 2 previous similar messages [ 2758.126708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2758.134372] Lustre: Skipped 3 previous similar messages [ 2758.137941] Lustre: 98652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2765.554832] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2766.490043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2766.510192] Lustre: Skipped 2 previous similar messages [ 2766.516852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2766.587266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2775.338188] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2779.963378] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2780.142358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2785.053810] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2785.295794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2804.436556] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2827.269521] Lustre: Failing over lustre-MDT0000 [ 2828.265868] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.273397] LustreError: 95279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.292269] Lustre: Skipped 4 previous similar messages [ 2828.321406] LustreError: 95279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2829.864324] Lustre: server umount lustre-MDT0000 complete [ 2844.648693] Lustre: 66691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788943110/real 1788943110] req@00000000a7918079 x1875841876503680/t0(0) o400->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788943126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2844.684454] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.703074] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2854.708206] LDISKFS-fs (dm-0): recovery complete [ 2854.716498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.253621] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2859.674880] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2860.313353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2860.315420] Lustre: 95277:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2860.526627] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2860.615767] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2860.953001] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2860.963052] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2860.976783] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2861.322714] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2861.335634] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2861.692567] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2861.699397] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2862.029117] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2862.052452] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 2 previous similar messages [ 2862.377644] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2862.392421] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2862.712766] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2862.727832] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2864.321073] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2864.332814] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 2864.670767] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2864.683072] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2865.000467] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2865.011119] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2868.512615] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2868.522070] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2868.850115] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2868.870305] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2869.195715] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2869.209409] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2876.513202] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2876.527830] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2876.852169] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2876.867300] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2877.505095] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2877.518343] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 25 previous similar messages [ 2892.702318] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2892.714315] Lustre: 101502:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 50 previous similar messages [ 2893.033093] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2893.060085] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2893.713062] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2893.725111] LustreError: 101502:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2920.032235] Lustre: 101502:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2920.044724] Lustre: 101502:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2920.055577] Lustre: 101502:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 293a88ac-5069-454b-864e-0651274b1298@192.168.204.21@tcp [ 2920.074615] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2920.083908] Lustre: 101502:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@0000000023c42d8a x1875841856380416/t0(8589934783) o101->293a88ac-5069-454b-864e-0651274b1298@192.168.204.21@tcp:157/0 lens 592/0 e 0 to 0 dl 1788943212 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2920.119081] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2920.129730] Lustre: lustre-MDT0000: Denying connection for new client 293a88ac-5069-454b-864e-0651274b1298 (at 192.168.204.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2920.152299] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2920.201196] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2920.213337] Lustre: Skipped 2 previous similar messages [ 2920.214216] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2920.250231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:49 to 0x2c0000401:545) [ 2920.252684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2936.289403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2936.305745] LustreError: Skipped 1 previous similar message [ 2936.312359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2936.321231] Lustre: Skipped 3 previous similar messages [ 2950.113687] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2950.194735] Lustre: server umount lustre-OST0000 complete [ 2956.769101] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.779240] Lustre: Skipped 4 previous similar messages [ 2957.798416] LustreError: 98225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.823751] LustreError: 98225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [ 2959.440584] Lustre: server umount lustre-OST0001 complete [ 2968.500285] Lustre: server umount lustre-MDT0000 complete [ 2970.928414] LustreError: 98651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943252 with bad export cookie 2591475699510364785 [ 2970.933275] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.933941] LustreError: 98651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2971.164236] Lustre: server umount lustre-MDT0001 complete [ 2977.557620] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 04:40:57 (1788943257) [ 2985.070366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.091655] Lustre: *** cfs_fail_loc=197, val=0*** [ 3000.800625] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3005.920799] LustreError: 103475:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 3011.138215] Lustre: *** cfs_fail_loc=197, val=0*** [ 3011.142736] Lustre: Skipped 65 previous similar messages [ 3031.713680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.722953] Lustre: Skipped 2 previous similar messages [ 3039.194636] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3047.303911] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3049.179275] Lustre: server umount lustre-OST0000 complete [ 3056.181405] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 04:42:15 (1788943335) [ 3075.631534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.862785] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3088.696992] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3091.846546] LustreError: 105125:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3091.853701] LustreError: 105125:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3091.859615] LustreError: 105125:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 3092.194226] Lustre: server umount lustre-MDT0000 complete [ 3124.567736] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3135.446478] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3175.915429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.299689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3176.372385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3176.507836] Lustre: lustre-MDT0000: new disk, initializing [ 3176.662347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3180.750357] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3190.339249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.447977] Lustre: 109303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3190.659222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3194.406167] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3203.124852] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3209.527600] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3216.185674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3216.506246] Lustre: 111273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3223.156421] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3223.371961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3232.804964] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3237.373944] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3237.640265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3241.631836] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3241.794468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3244.014019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.015519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.023708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3244.023773] Lustre: Skipped 14 previous similar messages [ 3244.031649] LustreError: Skipped 3 previous similar messages [ 3244.038786] Lustre: Skipped 3 previous similar messages [ 3249.657389] Lustre: server umount lustre-OST0000 complete [ 3259.364727] LustreError: 109310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.387678] LustreError: 109310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3259.421578] Lustre: server umount lustre-MDT0000 complete [ 3263.874440] LustreError: 109293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943545 with bad export cookie 2591475699510375509 [ 3263.883926] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.898876] LustreError: 109293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3264.373418] Lustre: server umount lustre-MDT0001 complete [ 3274.748795] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 04:45:53 (1788943553) [ 3284.657655] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3288.291155] Key type lgssc unregistered [ 3288.652958] LNet: 113759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.657203] LNetError: 113759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.665921] LNet: Removed LNI 192.168.204.121@tcp [ 3289.404204] Key type .llcrypt unregistered [ 3289.406695] Key type ._llcrypt unregistered [ 3305.900853] Key type ._llcrypt registered [ 3305.903992] Key type .llcrypt registered [ 3306.015629] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3320.830071] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3321.589449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3321.604242] alg: No test for adler32 (adler32-zlib) [ 3322.644928] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3322.890104] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3324.593434] Key type lgssc registered [ 3325.619343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3365.402442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3365.426642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.687039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3366.705105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3366.768136] Lustre: lustre-MDT0000: new disk, initializing [ 3366.852377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.877942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3371.280349] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3381.407028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3381.655511] Lustre: lustre-OST0000: new disk, initializing [ 3381.658795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3381.663194] Lustre: Skipped 1 previous similar message [ 3381.669603] Lustre: 117924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3381.775966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.846898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3382.869488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3382.952267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3388.000319] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3397.185696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3397.369191] Lustre: lustre-OST0001: new disk, initializing [ 3397.373254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3397.379596] Lustre: 118788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3397.452876] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3403.441520] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3406.431437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3406.445568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3406.475783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3411.430667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.434974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.440686] Lustre: Skipped 1 previous similar message [ 3411.457925] Lustre: Skipped 1 previous similar message [ 3416.552070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.573518] Lustre: Skipped 1 previous similar message [ 3417.223469] Lustre: server umount lustre-MDT0000 complete [ 3423.043167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3427.647307] LustreError: 117113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943709 with bad export cookie 6412976756790695270 [ 3427.655837] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.666973] LustreError: 117113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3427.778422] Lustre: server umount lustre-OST0000 complete [ 3431.384493] Lustre: server umount lustre-OST0001 complete [ 3439.589796] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 04:48:38 (1788943718) [ 3462.727719] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 04:49:01 (1788943741) [ 3474.562718] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3500.997030] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 04:49:40 (1788943780) [ 3539.070650] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3548.796715] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3580.190788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.402886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3580.431283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3580.489314] Lustre: lustre-MDT0000: new disk, initializing [ 3580.553921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.575849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3584.498550] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3595.007576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.141489] Lustre: 126541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3595.215900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3595.224238] Lustre: Skipped 1 previous similar message [ 3595.295544] Lustre: lustre-MDT0001: new disk, initializing [ 3595.416687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.473100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3595.493176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3599.782802] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3608.950353] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3614.893218] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3621.398654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.623755] Lustre: lustre-OST0000: new disk, initializing [ 3621.630302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3621.635237] Lustre: 128510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3621.707412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.340333] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3628.594790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3628.601623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3628.684286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3633.795872] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3638.079329] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3638.331630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3642.874201] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3643.095734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3648.997431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3649.007729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.026614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3649.510485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3651.521498] Lustre: server umount lustre-OST0000 complete [ 3657.186559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3657.199542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.217933] Lustre: Skipped 1 previous similar message [ 3657.236855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.261034] Lustre: Skipped 2 previous similar messages [ 3659.755718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.341934] Lustre: server umount lustre-MDT0000 complete [ 3664.348886] LustreError: 127346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943946 with bad export cookie 6412976756790696495 [ 3664.351235] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3664.368225] LustreError: 127346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3664.657530] Lustre: server umount lustre-MDT0001 complete [ 3673.472591] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 04:52:32 (1788943952) [ 3679.388223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.139408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.883970] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3685.347497] LustreError: 130921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.368331] LustreError: 130921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3689.448080] LustreError: 130920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.355686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.241542] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3705.741826] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3712.375617] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3719.768685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3720.125777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.140465] Lustre: Skipped 1 previous similar message [ 3726.711950] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3734.902875] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3742.946754] Lustre: *** cfs_fail_loc=159, val=0*** [ 3758.422296] Lustre: lustre-MDT0000: Client 19114000-328a-4b8e-8405-d441b1cbbbe6 (at 192.168.204.21@tcp) reconnecting [ 3760.412938] Lustre: *** cfs_fail_loc=159, val=0*** [ 3760.432705] Lustre: Skipped 5 previous similar messages [ 3776.896188] Lustre: lustre-MDT0000: Client 19114000-328a-4b8e-8405-d441b1cbbbe6 (at 192.168.204.21@tcp) reconnecting [ 3794.913281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3794.920896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.934919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3794.948239] Lustre: Skipped 1 previous similar message [ 3794.968396] Lustre: Skipped 2 previous similar messages [ 3799.701151] Lustre: server umount lustre-OST0000 complete [ 3805.168348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3805.172270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.187475] Lustre: Skipped 1 previous similar message [ 3810.154368] Lustre: server umount lustre-MDT0000 complete [ 3810.298645] LustreError: 133521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.312104] LustreError: 133521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3814.066605] LustreError: 130902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944095 with bad export cookie 6412976756790697377 [ 3814.076640] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.420118] Lustre: server umount lustre-MDT0001 complete [ 3825.773388] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3828.910161] Key type lgssc unregistered [ 3829.268871] LNet: 134702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3829.278088] LNetError: 134702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3829.306959] LNet: Removed LNI 192.168.204.121@tcp [ 3830.285845] Key type .llcrypt unregistered [ 3830.287966] Key type ._llcrypt unregistered [ 3855.072937] Key type ._llcrypt registered [ 3855.074427] Key type .llcrypt registered [ 3855.215478] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3856.466644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3856.637838] alg: No test for adler32 (adler32-zlib) [ 3857.770200] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 3857.949509] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3859.592182] Key type lgssc registered [ 3860.787888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3867.805648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3867.824140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.439725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.411289] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3879.906666] LustreError: 136112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.937969] LustreError: 136112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3884.521807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.043788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3890.374446] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3898.474178] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3905.128509] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3911.988721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3912.378043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.433864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3918.862440] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3927.112586] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3939.396605] Lustre: *** cfs_fail_loc=159, val=0*** [ 3956.050136] Lustre: lustre-MDT0000: Client 9ea03ee2-5ad2-4971-ad89-6e86370321b6 (at 192.168.204.21@tcp) reconnecting [ 3959.008960] Lustre: *** cfs_fail_loc=159, val=0*** [ 3974.484995] Lustre: lustre-MDT0001: Client 9ea03ee2-5ad2-4971-ad89-6e86370321b6 (at 192.168.204.21@tcp) reconnecting [ 3976.469073] Lustre: *** cfs_fail_loc=159, val=0*** [ 3976.473644] Lustre: Skipped 6 previous similar messages [ 3992.905658] Lustre: lustre-MDT0001: Client 9ea03ee2-5ad2-4971-ad89-6e86370321b6 (at 192.168.204.21@tcp) reconnecting [ 4008.843739] Lustre: 139173:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4012.607205] Lustre: *** cfs_fail_loc=159, val=0*** [ 4012.619284] Lustre: Skipped 1 previous similar message [ 4029.288856] Lustre: lustre-MDT0000: Client 7f575171-8060-4d76-a07b-2f6264e19c53 (at 192.168.204.21@tcp) reconnecting [ 4031.075967] Lustre: *** cfs_fail_loc=159, val=0*** [ 4031.081894] Lustre: Skipped 13 previous similar messages [ 4046.704428] Lustre: lustre-MDT0000: Client 7f575171-8060-4d76-a07b-2f6264e19c53 (at 192.168.204.21@tcp) reconnecting [ 4048.159662] Lustre: 139466:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4068.336247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4068.343655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.355262] LustreError: Skipped 1 previous similar message [ 4068.355978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4068.388679] Lustre: Skipped 1 previous similar message [ 4068.427107] Lustre: Skipped 1 previous similar message [ 4073.451198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4073.462221] Lustre: Skipped 1 previous similar message [ 4074.325225] Lustre: server umount lustre-OST0000 complete [ 4079.587194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4079.595051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.612402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.684316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.689652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.203629] Lustre: server umount lustre-MDT0000 complete [ 4088.802667] LustreError: 136109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.828878] LustreError: 136109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4089.166124] LustreError: 136093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944371 with bad export cookie 17959219648870563938 [ 4089.173675] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.182668] LustreError: 136093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4089.556431] Lustre: server umount lustre-MDT0001 complete [ 4101.562210] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4104.338101] Key type lgssc unregistered [ 4104.601351] LNet: 140552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4104.609248] LNetError: 140552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4104.625285] LNet: Removed LNI 192.168.204.121@tcp [ 4105.563791] Key type .llcrypt unregistered [ 4105.565757] Key type ._llcrypt unregistered [ 4131.680969] Key type ._llcrypt registered [ 4131.691476] Key type .llcrypt registered [ 4131.883279] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4133.018458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4133.113123] alg: No test for adler32 (adler32-zlib) [ 4134.243208] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4134.518303] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4136.312385] Key type lgssc registered [ 4137.384597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4145.848321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4145.866725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.384251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.802790] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4157.931371] LustreError: 141971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.949385] LustreError: 141971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4160.703419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.091930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4165.977641] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4174.312128] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.736944] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4187.809773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4188.046599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.180898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4193.526764] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4201.295775] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4205.522430] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4211.540892] Lustre: 144564:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4215.057775] Lustre: 144614:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4216.821372] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4217.911060] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4221.844517] Lustre: 144760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4223.027311] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4223.873180] Lustre: 144858:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4231.139538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.162455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.167193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.197881] Lustre: Skipped 1 previous similar message [ 4234.438346] Lustre: server umount lustre-OST0000 complete [ 4241.382489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.396843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.399988] Lustre: Skipped 2 previous similar messages [ 4241.419197] Lustre: Skipped 1 previous similar message [ 4245.009817] Lustre: server umount lustre-MDT0000 complete [ 4246.497605] LustreError: 143824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.520599] LustreError: 143824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4248.700971] LustreError: 141952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944530 with bad export cookie 8575969278436194200 [ 4248.714044] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.719852] LustreError: 141952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4249.059801] Lustre: server umount lustre-MDT0001 complete [ 4261.063527] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4263.966222] Key type lgssc unregistered [ 4264.325774] LNet: 145917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4264.334718] LNetError: 145917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4265.386080] LNet: Removed LNI 192.168.204.121@tcp [ 4266.437835] Key type .llcrypt unregistered [ 4266.443125] Key type ._llcrypt unregistered [ 4289.545835] Key type ._llcrypt registered [ 4289.551037] Key type .llcrypt registered [ 4289.695082] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4290.879558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4290.945334] alg: No test for adler32 (adler32-zlib) [ 4291.995230] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4292.250213] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4293.936567] Key type lgssc registered [ 4295.042480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4302.218079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4302.231387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.801108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.859780] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4314.082191] LustreError: 147329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.102088] LustreError: 147329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4315.644791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.013904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4320.257349] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4327.314505] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4332.298771] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4338.535337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.905718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.975605] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4345.035325] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4348.213171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4351.910511] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4358.724125] Lustre: *** cfs_fail_loc=159, val=0*** [ 4375.403532] Lustre: lustre-MDT0000: Client b48c1a3b-b049-46cc-886e-d69acafbe368 (at 192.168.204.21@tcp) reconnecting [ 4381.160737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4381.164608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.172994] LustreError: Skipped 1 previous similar message [ 4381.186412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4382.178208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4382.193085] LustreError: Skipped 6 previous similar messages [ 4382.207992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.229920] LustreError: 149156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.419212] Lustre: server umount lustre-OST0000 complete [ 4387.812304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4387.824861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.846223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.347285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.355670] Lustre: Skipped 1 previous similar message [ 4392.376081] Lustre: server umount lustre-MDT0000 complete [ 4394.466394] LustreError: 149829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.492522] LustreError: 149829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4395.573909] LustreError: 147309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944677 with bad export cookie 10588641186612814562 [ 4395.581903] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.583281] LustreError: 147309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4395.921030] Lustre: server umount lustre-MDT0001 complete [ 4405.905703] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4408.864835] Key type lgssc unregistered [ 4409.119390] LNet: 150991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4409.125946] LNetError: 150991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4409.142587] LNet: Removed LNI 192.168.204.121@tcp [ 4409.912706] Key type .llcrypt unregistered [ 4409.914921] Key type ._llcrypt unregistered [ 4430.662218] Key type ._llcrypt registered [ 4430.665097] Key type .llcrypt registered [ 4430.757518] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4431.537169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4431.597045] alg: No test for adler32 (adler32-zlib) [ 4432.642911] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4432.871290] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4434.528194] Key type lgssc registered [ 4435.467941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4442.682758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4442.714460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.133366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.206610] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4454.370033] LustreError: 152417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.390460] LustreError: 152417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4456.975451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.437382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.278356] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4468.372280] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4473.266359] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4478.931388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.320375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.433865] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4485.698990] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4489.594180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4493.244205] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4513.160674] Lustre: lustre-MDT0001: evicting c814871a-39a5-486b-bcbf-9e7a0d451c8a (at 192.168.204.21@tcp) by administrative request [ 4513.169021] Lustre: Skipped 1 previous similar message [ 4543.969853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.975118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.989056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4546.017306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4546.022414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4546.022601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.027219] Lustre: Skipped 1 previous similar message [ 4547.286526] Lustre: server umount lustre-OST0000 complete [ 4551.139414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.143917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4551.149328] Lustre: Skipped 1 previous similar message [ 4556.260036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.261783] LustreError: 152413:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.265817] Lustre: Skipped 1 previous similar message [ 4556.371652] Lustre: server umount lustre-MDT0000 complete [ 4560.064769] LustreError: 152398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944841 with bad export cookie 17737491805335106016 [ 4560.073987] LustreError: 152398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4560.077807] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4560.394266] Lustre: server umount lustre-MDT0001 complete [ 4571.608271] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4574.898784] Key type lgssc unregistered [ 4575.246723] LNet: 156488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4575.256822] LNetError: 156488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4576.295258] LNet: Removed LNI 192.168.204.121@tcp [ 4577.237494] Key type .llcrypt unregistered [ 4577.241317] Key type ._llcrypt unregistered [ 4642.986376] Key type ._llcrypt registered [ 4642.992298] Key type .llcrypt registered [ 4643.084808] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4643.782897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4643.881143] alg: No test for adler32 (adler32-zlib) [ 4644.894664] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4645.151256] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4646.849791] Key type lgssc registered [ 4647.964934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4654.756712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4654.784917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.258228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4660.038341] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4666.852832] LustreError: 158966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.885087] LustreError: 158966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4669.605140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.017364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4674.881500] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4683.288117] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4687.912590] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4693.738274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.979411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4695.028467] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4699.857093] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4701.174677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4707.062820] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4715.489809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4715.499992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.517824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4716.515622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.519372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4716.524186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4720.289925] Lustre: server umount lustre-OST0000 complete [ 4726.243463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4726.248956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.258265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.265952] Lustre: Skipped 1 previous similar message [ 4729.915129] Lustre: server umount lustre-MDT0000 complete [ 4731.875507] LustreError: 158965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.893219] LustreError: 158965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4733.472232] LustreError: 158946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945015 with bad export cookie 15261964631990530289 [ 4733.483507] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.487272] LustreError: 158946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4733.953054] Lustre: server umount lustre-MDT0001 complete [ 4746.460709] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4749.507725] Key type lgssc unregistered [ 4749.847900] LNet: 162508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4749.855820] LNetError: 162508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4750.892777] LNet: Removed LNI 192.168.204.121@tcp [ 4751.993780] Key type .llcrypt unregistered [ 4751.996299] Key type ._llcrypt unregistered [ 4778.050818] Key type ._llcrypt registered [ 4778.054755] Key type .llcrypt registered [ 4778.164607] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4778.835608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4778.844705] alg: No test for adler32 (adler32-zlib) [ 4779.955123] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4780.190367] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4781.864183] Key type lgssc registered [ 4782.922788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4796.656417] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 05:11:15 (1788945075) [ 4811.537525] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 4819.736191] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4856.709515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4856.746610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.068933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4858.127908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4858.235705] Lustre: lustre-MDT0000: new disk, initializing [ 4858.401416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4858.453318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4863.562710] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4873.946905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.077893] Lustre: 166769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4874.131201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4874.133827] Lustre: Skipped 1 previous similar message [ 4874.205563] Lustre: lustre-MDT0001: new disk, initializing [ 4874.339713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4874.384168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4874.398716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4879.032985] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4887.930797] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4894.297533] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4900.565966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4900.865802] Lustre: lustre-OST0000: new disk, initializing [ 4900.874852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4900.893861] Lustre: 168737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4900.986940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.554255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4906.572873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4906.736711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4907.538695] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4917.042457] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4927.457437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4927.463108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.467415] LustreError: Skipped 1 previous similar message [ 4927.470396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4927.492867] Lustre: Skipped 1 previous similar message [ 4932.580401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4932.590595] Lustre: Skipped 2 previous similar messages [ 4932.952398] Lustre: server umount lustre-OST0000 complete [ 4937.701626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.726515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4937.741227] Lustre: Skipped 1 previous similar message [ 4943.007264] Lustre: server umount lustre-MDT0000 complete [ 4947.296177] LustreError: 168736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945229 with bad export cookie 10297359494001024427 [ 4947.298089] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.307088] LustreError: 168736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4947.689230] Lustre: server umount lustre-MDT0001 complete [ 4957.596451] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 4960.182642] Key type lgssc unregistered [ 4960.541845] LNet: 170641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4960.552058] LNetError: 170641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4960.567296] LNet: Removed LNI 192.168.204.121@tcp [ 4961.269737] Key type .llcrypt unregistered [ 4961.273064] Key type ._llcrypt unregistered [ 4982.761712] Key type ._llcrypt registered [ 4982.763890] Key type .llcrypt registered [ 4982.858126] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4983.571608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4983.653216] alg: No test for adler32 (adler32-zlib) [ 4984.844964] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 4985.084073] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 4986.744541] Key type lgssc registered [ 4987.783088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4995.836390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4995.875306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.548839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.209248] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5007.846485] LustreError: 172051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.866760] LustreError: 172051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5010.163116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.586417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5015.088174] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5022.267581] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5028.425630] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5034.815494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5035.073537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.708801] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 5042.232859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5049.091558] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5055.085228] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5055.106572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.134555] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5055.141617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.158299] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5056.102967] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5056.128462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.145176] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5056.158269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.185994] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5057.150153] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5057.186140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.200759] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5057.206105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.227400] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5058.074300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.911475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.843177] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5059.852457] Lustre: Skipped 2 previous similar messages [ 5059.883772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.893657] Lustre: Skipped 2 previous similar messages [ 5059.898139] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5059.903251] Lustre: Skipped 2 previous similar messages [ 5059.907292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.918173] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5059.930455] Lustre: Skipped 2 previous similar messages [ 5061.119232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.255053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.225209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.300270] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5064.306271] Lustre: Skipped 3 previous similar messages [ 5064.320988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.330413] Lustre: Skipped 3 previous similar messages [ 5064.336371] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5064.342697] Lustre: Skipped 3 previous similar messages [ 5064.348890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.360794] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5064.368524] Lustre: Skipped 3 previous similar messages [ 5065.450156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.606717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.496815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.458712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.344845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.280817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.134245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.051173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.811773] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5072.819714] Lustre: Skipped 8 previous similar messages [ 5072.869483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.885600] Lustre: Skipped 8 previous similar messages [ 5072.894128] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5072.900260] Lustre: Skipped 8 previous similar messages [ 5072.906653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.921769] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5072.929276] Lustre: Skipped 8 previous similar messages [ 5073.821743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.756388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.791858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.744649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.601077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.373413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.295255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.225686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.006558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.753161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.523266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.243920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.992163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.828150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.648229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.481742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.317775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.998996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.783956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.524939] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5089.530731] Lustre: Skipped 19 previous similar messages [ 5089.547339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.565391] Lustre: Skipped 19 previous similar messages [ 5089.576314] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5089.582403] Lustre: Skipped 19 previous similar messages [ 5089.589548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.601094] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5089.607757] Lustre: Skipped 19 previous similar messages [ 5090.448032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.298987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.917973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.867735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.763131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.787219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.619152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.422899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.053884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.813854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.739686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.612578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.401198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.255556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.146263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.934329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.908686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.946341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.920744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.965207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.799757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.808901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.561411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.332236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.536968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.482558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.220204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.077768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.134123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.031580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.307302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.649250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.640117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.512853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.431399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.320953] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5122.331947] Lustre: Skipped 35 previous similar messages [ 5122.358545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.369663] Lustre: Skipped 35 previous similar messages [ 5122.374660] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5122.383977] Lustre: Skipped 35 previous similar messages [ 5122.389054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.398474] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5122.412539] Lustre: Skipped 35 previous similar messages [ 5123.183454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.168954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.051948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.983737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.843306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.704232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.561956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.423916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.281884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.174131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.919737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.643578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.349981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.120197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.835493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.578409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.482438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.260150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.042543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.920472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.670815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.432492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.346870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.378603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.254430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.093893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.891319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.704855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.576883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.430441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.192495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.989840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.815690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.745294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.603992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.413756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.284505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.024712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.869275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.708959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.593092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.383678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.139646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.929029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.684328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.372289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.093862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.716957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.418593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.065907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.893176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.563678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.270389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.971543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.655085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.308502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.013372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.767490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.529335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.277425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.031157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.824621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.568404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.215747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.945281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.656223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.367143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.208155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.994274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.784255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.775848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.713383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.643535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.451565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.336866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.071469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.795302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.505748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.200845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.949693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.584958] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5186.591015] Lustre: Skipped 80 previous similar messages [ 5186.599620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.607740] Lustre: Skipped 80 previous similar messages [ 5186.612474] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5186.615781] Lustre: Skipped 80 previous similar messages [ 5186.618789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.628463] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5186.632952] Lustre: Skipped 80 previous similar messages [ 5187.293905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.993886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.659963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.348119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.996300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.536915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.217099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.003441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.735954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.590246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.345598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.328969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.078934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.832524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.736495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.758132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.551543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.481571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.416172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.341835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.146347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.052364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.945172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.755967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.581291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.563895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.454487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.346855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.150652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.259863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.244751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.261083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.313785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.314104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.395054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.235573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.263068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.192864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.185571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.996395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.848359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.768792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.726619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.473633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.334478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.060697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.075513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.910592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.726283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.445180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.186483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.202927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.140832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.186818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.260909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.029300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.951753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.910525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.907396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.778992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.696892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.520180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.453028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.273483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.084648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.990105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.895397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.622196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.515622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.203164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.928801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.604275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.447524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.346827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.180147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.095966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.969976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.914477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.668582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.520327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.401588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.244856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.085396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.935944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.743806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.537299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.407489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.247594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.020062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.683815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.343090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.018990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.741664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.436092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.132335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.836488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.651390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.460825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.141224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.855410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.614783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.365519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.186728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.956720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.809230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.725049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.628781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.620177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.566497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.408286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.392759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.346288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.378724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.317249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.318768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.189578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.967769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.917055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.721499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.512604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.289601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.172461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.023274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.706914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.523217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.380945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.208740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.979501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.699514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.321440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.045449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.741031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.547117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.281213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.005529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.665981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.455510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.179714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.980076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.668158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.451975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.277835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.215742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.957674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.774033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.715306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.573722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.501487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.355724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.193421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.133470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.918382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.816721] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5314.821482] Lustre: Skipped 152 previous similar messages [ 5314.837096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.853162] Lustre: Skipped 152 previous similar messages [ 5314.857805] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5314.863880] Lustre: Skipped 152 previous similar messages [ 5314.869884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.881598] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5314.887932] Lustre: Skipped 152 previous similar messages [ 5315.685451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.608548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.468884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.198182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.993780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.742688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.558730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.534931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.353235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.050067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.785567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.814599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.674186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.639159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.522567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.239357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.921384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.814096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.660739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.550623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.427577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.301156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.132632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.931861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.974981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.783642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.660936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.458478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.223989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.080580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.991607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.937688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.895599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.824792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.810911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.694554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.543947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.255978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.065470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.968958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.019562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.967045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.863890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.771161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.762393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.426423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.207411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.020718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.861413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.702520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.571461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.383904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.333625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.097670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.909621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.722839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.608166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.358781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.201169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.015857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.907142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.832549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.614527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.399909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.197490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.867644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.604297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.284664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.992611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.684238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.470421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.227392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.030947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.858457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.561058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.291894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.123985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.917338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.696315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.432249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.299770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.091908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.904380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.675706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.490864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.262326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.977746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.730069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.401367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.090343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.802772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.484613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.170995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.821811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.568090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.195604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.937059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.681305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.469547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.282528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.128735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.958947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.642699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.422461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.148309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.794663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.498648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.196567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.887223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.539314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.241940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.878468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.539976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.167723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.845526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.416408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.057793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.752674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.416881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.099531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.725485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.334069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.004824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.650664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.343456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.954626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.633187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.334963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.002884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.608619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.203911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.910532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.540260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.212565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.845847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.541732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.248722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.906681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.647897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.490625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.221477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.952270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.721501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.383936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.115916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.814511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.520491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.237271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.014348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.732814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.546083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.410990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.167594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.959369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.753916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.453830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.214809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.897248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.523125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.224260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.917568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.698064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.358925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.114151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.823232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.494613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.045664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.726295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.425071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.184549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.974891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.623593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.146095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.856910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.710422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.727118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.761488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.668798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.455707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.313463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.231790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.439741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.644640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.853671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.727328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.664440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.548973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.527128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.658186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.598708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.570464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.823516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.041169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.816976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.884330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.960396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.966823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.913736] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5471.932415] LustreError: 172071:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5471.942059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.708662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.681587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.540942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.434134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.304414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.190939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.189838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.166436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.164229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.010443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.826196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.553315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.399054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.133952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.917574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.617986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.393646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.220215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.948318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.731808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.481602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.337090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.073000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.759551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.591876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.319089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.181627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.884372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.702412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.429629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.185500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.859885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.721813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.626347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.462079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.178558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.962719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.586096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.232719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.941184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.723256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.531751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.279386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.897857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.480698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.083891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.750542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.407071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.304788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.053557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.835865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.565767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.344774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.100251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.010877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.709280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.609015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.594097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.554422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.590707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.864249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.870251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.218165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.466617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.549652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.543050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.662551] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5527.676177] LustreError: 172071:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5527.687976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.690453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.603211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.545128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.645948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.505631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.481904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.302263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.125037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.893462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.538109] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5536.550134] LustreError: 172071:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5536.551880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.281344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.103088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.873568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.595247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.258489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.062583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.759713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.632417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.397171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.229673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.906887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.608630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.427667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.296466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.272112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.222588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.117862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.986016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.839688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.807284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.637117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.458845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.277398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.981579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.825682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.570530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.316839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.028325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.795565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.565570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.462277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.301898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.062985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.798991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.617652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.527052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.642797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.644189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.732245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.622480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.517272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.278521] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5571.286490] Lustre: Skipped 316 previous similar messages [ 5571.304903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.313557] Lustre: Skipped 316 previous similar messages [ 5571.317311] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5571.320824] Lustre: Skipped 316 previous similar messages [ 5571.322938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.329512] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5571.333058] Lustre: Skipped 316 previous similar messages [ 5572.162913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.957709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.716420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.470402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.377719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.996255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.670328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.368877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.231622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.099367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.105497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.763654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.798880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.616526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.459256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.290276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.301151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.309037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.384286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.242379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.876966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.707587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.590253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.803113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.796622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.586231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.357397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.184179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.813573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.806731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.688361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.918985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.800058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.706418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.672392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.492244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.242606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.099714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.071068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.926321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.780934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.650265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.488940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.409128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.315731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.106607] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5611.121065] LustreError: 172071:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5611.125903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.051425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.960849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.809891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.630491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.565730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.621776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.557825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.322637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.093423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.958196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.740457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.695511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.644480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.507732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.292206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.096825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.919221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.727852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.705326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.627900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.422290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.208212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.184280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.074507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.040655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.877124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.642947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.383086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.129949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.027029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.841382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.873952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.837276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.107842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.377478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.228204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.271053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.552121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.672117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.674520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.512812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.482180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.418134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.516975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.171345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.297588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.447746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.518586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.325579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.190448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.071809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.890200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.690432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.421713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.238962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.926271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.630665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.317195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.281654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.088973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.803083] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5665.812375] LustreError: 172071:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5665.826679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.556343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.207626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.991629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.781798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.512112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.259495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.013630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.816381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.518233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.277849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.088068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.772987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.511908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.307295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.983300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.792085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.558021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.379930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.173203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.993724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.667077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.419591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.128985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.682236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.375451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.126364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.822288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.751762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.493386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.295392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.974887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.757028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.397776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.278273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.163583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.007565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.766717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.625745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.399775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.290893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.191948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.988736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.845890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.589866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.405086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.071809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.114717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.049866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.265735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.373367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.190743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.357707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.566607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.575432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.904408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.087254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.315946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.441417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.367887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.137833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.796962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.775891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.885531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.006730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.020695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.955881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.657656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.551075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.480443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.333460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.098908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.027133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.777996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.575370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.302195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.109299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.856667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.636466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.407939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.320462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.047545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.931360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.886776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.052590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.795230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.683000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.631533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.709719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.785943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.570167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.500414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.253456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.148197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.089812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.936699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.717577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.823816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.890419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.014225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.961521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.708589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.737560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.001831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.268932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.485902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.583571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.670634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.494659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.699632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.879177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.943682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.295706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.230846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.127295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.988983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.799247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.040685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.066505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.104230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.971406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.083857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.842319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.225300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.486507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.385538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.685257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.608710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.846302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.754430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.897748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.728876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.717539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.693326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.617884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.831196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.869194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.825246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.779609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.852867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.750327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.702770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.734918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.878703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.689240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.644774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.491159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.421155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.437859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.548244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.522376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.448816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.311252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.172063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.042678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.755224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.563073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.276977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.104581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.096318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.020647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.980848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.752936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.707012] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.518838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.321632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.115974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.863608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.669590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.434226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.153954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.060430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.998185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.930771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.915936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.844460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.625314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.525853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.456297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.325450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.341149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.196527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.034684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.820124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.612689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.398260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.144713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.916172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.639443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.332785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.059210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.755339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.410103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.133147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.832606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.635804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.351270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.195659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.867409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.677455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.418929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.158810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.821795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.566488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.382389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.106856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.677856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.406263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.132142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.942891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.711399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.490586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.276617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.122841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.002404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.904917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.854292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.664372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.656444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.501917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.471282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.318206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.223593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.054900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.842273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.587381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.259184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.026180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.811902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.667817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.506268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.450065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.229480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.081829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.971868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.963798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.840602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.722805] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5874.732544] LustreError: 172071:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5874.736912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.597519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.521502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.423468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.323844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.233403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.025770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.846616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.728357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.606653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.371743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.185681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.095239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.022623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.876286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.721691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.613545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.440919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.411544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.198435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.115494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.019917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.830854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.692806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.639775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.525414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.472779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.369444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.335354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.261256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.225215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.914469] LustreError: 173957:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6266.932488] LustreError: 172071:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6909.923840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6909.933394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.946255] Lustre: Skipped 374 previous similar messages [ 6909.952039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6911.974647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6911.991225] Lustre: Skipped 1 previous similar message [ 6916.440827] Lustre: server umount lustre-OST0000 complete [ 6922.213980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6922.220115] Lustre: Skipped 1 previous similar message [ 6927.345476] LustreError: 173323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.374479] LustreError: 173323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6927.616825] Lustre: server umount lustre-MDT0000 complete [ 6932.455958] LustreError: 172779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.471725] LustreError: 172779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6933.154317] LustreError: 172032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947215 with bad export cookie 2057396034085173877 [ 6933.164899] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.178663] LustreError: 172032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6933.567638] Lustre: server umount lustre-MDT0001 complete [ 6947.063645] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 6950.600518] Key type lgssc unregistered [ 6951.032093] LNet: 227283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6951.055301] LNetError: 227283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6951.084761] LNet: Removed LNI 192.168.204.121@tcp [ 6952.299941] Key type .llcrypt unregistered [ 6952.303296] Key type ._llcrypt unregistered [ 6981.149186] Key type ._llcrypt registered [ 6981.152060] Key type .llcrypt registered [ 6981.295606] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 6982.842027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6982.922318] alg: No test for adler32 (adler32-zlib) [ 6984.097813] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 6984.427067] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 6986.192500] Key type lgssc registered [ 6987.405496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6995.434285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6995.448783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.067199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7002.385946] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7006.204249] LustreError: 228696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.220938] LustreError: 228696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7011.309108] LustreError: 228697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.251884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.572481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7017.140468] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7024.504619] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7030.218747] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7036.653738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7037.002957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7041.156764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7043.334655] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7051.401648] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7057.192152] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7058.977791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.999431] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7059.016908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7059.043129] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7059.043604] LustreError: 228716:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7061.988900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7061.997718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7061.998452] Lustre: Skipped 1 previous similar message [ 7062.010211] Lustre: Skipped 1 previous similar message [ 7064.375769] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 7067.108380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7069.468209] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 7069.475201] Lustre: Skipped 1 previous similar message [ 7074.579716] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 7074.597265] Lustre: Skipped 2 previous similar messages [ 7075.296503] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7075.436874] Lustre: server umount lustre-OST0000 complete [ 7079.113372] LustreError: 228716:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7110.729069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7111.185086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7118.455317] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7126.748757] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7127.590948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7127.617119] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7127.617782] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7127.638486] Lustre: Skipped 1 previous similar message [ 7132.024542] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7138.664781] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f0cd375d000.ost_server_uuid 50 [ 7140.605447] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f0cd375d000.ost_server_uuid in FULL state after 0 sec [ 7145.960272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7145.976507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.997406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7147.970397] Lustre: server umount lustre-OST0000 complete [ 7153.122067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.146538] Lustre: Skipped 1 previous similar message [ 7158.891527] Lustre: server umount lustre-MDT0000 complete [ 7163.264310] LustreError: 231930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947445 with bad export cookie 13671584781127966156 [ 7163.265886] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.287432] LustreError: 231930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7173.752783] Lustre: server umount lustre-MDT0001 complete [ 7186.993845] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 7190.351671] Key type lgssc unregistered [ 7190.676669] LNet: 233908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7190.683992] LNetError: 233908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7190.709823] LNet: Removed LNI 192.168.204.121@tcp [ 7191.726120] Key type .llcrypt unregistered [ 7191.729688] Key type ._llcrypt unregistered [ 7228.690768] Key type ._llcrypt registered [ 7228.697690] Key type .llcrypt registered [ 7228.827425] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7230.485129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7230.596221] alg: No test for adler32 (adler32-zlib) [ 7232.165660] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 7232.714784] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 7234.512422] Key type lgssc registered [ 7236.237877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7247.547468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7247.570850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.237352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.523815] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7259.623291] LustreError: 235635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.663343] LustreError: 235635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7264.740577] LustreError: 235634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7267.051388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.402697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7272.691889] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7276.473237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7284.888678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.388977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.931595] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7294.612747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7304.565861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7304.759398] Lustre: lustre-OST0001: new disk, initializing [ 7304.767686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7304.775263] Lustre: 237942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7304.916503] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7312.150415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7312.166383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7312.364923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7313.062926] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7326.151377] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7332.141440] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7348.196624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.208187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.214834] Lustre: Skipped 3 previous similar messages [ 7348.223774] Lustre: Skipped 3 previous similar messages [ 7352.846792] Lustre: server umount lustre-MDT0000 complete [ 7353.319535] LustreError: 235630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.350421] LustreError: 235630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7356.870363] LustreError: 238765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947638 with bad export cookie 11772373398836886175 [ 7356.870711] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7356.886054] LustreError: 238765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7357.248467] Lustre: server umount lustre-MDT0001 complete [ 7371.235349] Lustre: server umount lustre-OST0000 complete [ 7374.625095] Lustre: 234897:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788947640/real 1788947640] req@ffff98998acfb480 x1875847428540160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788947656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7374.659819] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.102575] Lustre: server umount lustre-OST0001 complete [ 7400.267638] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7412.683893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.355762] LustreError: 241057:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.472951] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.854904] LustreError: 241058:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.921734] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7428.689718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.097322] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7435.005831] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7438.179601] Lustre: 242202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7445.497613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.853448] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7452.530508] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7454.062017] LustreError: 242555:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7454.087572] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7454.092677] LustreError: 242555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7458.170316] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7462.383080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.828708] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7479.060234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7483.390888] Lustre: 244073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7495.650202] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7495.654799] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.664019] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7498.722728] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7498.741061] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7498.758887] Lustre: Skipped 2 previous similar messages [ 7501.928388] Lustre: server umount mylustre-MDT0000 complete [ 7503.846704] LustreError: 241058:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.889432] LustreError: 241058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7506.987598] LustreError: 244275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947788 with bad export cookie 11772373398836889038 [ 7507.035394] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7508.153854] Lustre: server umount mylustre-MDT0001 complete [ 7522.718222] Lustre: server umount mylustre-OST0000 complete [ 7525.088103] Lustre: 234898:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788947790/real 1788947790] req@ffff98999c408e00 x1875847428605824/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788947806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7525.125815] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.711541] Lustre: server umount mylustre-OST0001 complete [ 7553.863729] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7565.186257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.625450] LustreError: 246384:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.700718] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.704299] Lustre: Skipped 1 previous similar message [ 7570.820661] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7583.524508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.781912] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7594.481198] Lustre: 247526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7601.847852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.520087] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7608.743640] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7610.621399] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7618.417949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7625.345923] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7633.845958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7638.105122] Lustre: 249400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7649.775234] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.778494] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7649.792643] Lustre: Skipped 2 previous similar messages [ 7649.797665] Lustre: Skipped 4 previous similar messages [ 7652.160527] Lustre: server umount tfs-MDT0000 complete [ 7654.889331] LustreError: 248217:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.908533] LustreError: 248217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7656.229507] LustreError: 249402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947938 with bad export cookie 11772373398836890991 [ 7656.241851] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7656.243038] LustreError: 249402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7656.531902] Lustre: server umount tfs-MDT0001 complete [ 7669.414498] Lustre: server umount tfs-OST0000 complete [ 7684.909589] Lustre: server umount tfs-OST0001 complete [ 7712.101200] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7725.142380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7725.654399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.661674] Lustre: Skipped 3 previous similar messages [ 7730.597814] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7740.557833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.905288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7740.912659] Lustre: Skipped 2 previous similar messages [ 7745.570822] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7748.769574] Lustre: 252792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7756.694482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7765.400162] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7766.020902] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7770.557240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7777.403197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.431621] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7793.627676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7798.009091] Lustre: 254675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7805.722930] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 06:01:24 (1788948084) [ 7822.817879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7822.831120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.844694] Lustre: Skipped 1 previous similar message [ 7822.858544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7828.483446] Lustre: server umount lustre-MDT0000 complete [ 7832.042122] LustreError: 251653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.065027] LustreError: 251653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7832.853243] LustreError: 251629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948114 with bad export cookie 11772373398836892685 [ 7832.859309] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7832.870929] LustreError: 251629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7842.273571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7842.288571] Lustre: Skipped 7 previous similar messages [ 7847.392863] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7847.826470] Lustre: server umount lustre-MDT0001 complete [ 7858.647500] Lustre: server umount lustre-OST0000 complete [ 7869.574795] Lustre: server umount lustre-OST0001 complete [ 7877.956695] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 7889.899416] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7930.102991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7930.461328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7930.505513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7930.613462] Lustre: lustre-MDT0000: new disk, initializing [ 7930.813708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7930.823776] Lustre: Skipped 3 previous similar messages [ 7930.864283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7936.205402] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7948.619944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.844944] Lustre: 257860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7948.919797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7948.924497] Lustre: Skipped 1 previous similar message [ 7949.072662] Lustre: lustre-MDT0001: new disk, initializing [ 7949.361441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7949.384512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7956.028637] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7968.665353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7969.039990] Lustre: lustre-OST0000: new disk, initializing [ 7969.045517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7969.052511] Lustre: 259449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7971.089213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7971.115620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7971.315699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7976.272229] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7988.172542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7988.363814] Lustre: lustre-OST0001: new disk, initializing [ 7988.368307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7988.380438] Lustre: 260321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7989.576351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7989.584986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7989.703166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7995.205155] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8007.530243] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 06:04:46 (1788948286) [ 8022.788814] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 06:05:02 (1788948302) [ 8037.859568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8037.865631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.880322] Lustre: Skipped 5 previous similar messages [ 8037.886529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8037.895329] Lustre: Skipped 3 previous similar messages [ 8043.589653] Lustre: server umount lustre-OST0000 complete [ 8053.397024] Lustre: server umount lustre-MDT0000 complete [ 8057.199021] LustreError: 257854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948339 with bad export cookie 11772373398836894239 [ 8057.201799] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8057.216757] LustreError: 257854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8063.817285] Lustre: server umount lustre-MDT0001 complete [ 8075.176131] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8149.936905] Lustre: server umount lustre-OST0001 complete [ 8156.979879] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 8174.200592] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8217.756331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.020284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8218.053843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8218.139840] Lustre: lustre-MDT0000: new disk, initializing [ 8218.199499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8218.205465] Lustre: Skipped 3 previous similar messages [ 8218.218654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8223.160807] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8233.626873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.779652] Lustre: 265172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8233.844601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8233.848463] Lustre: Skipped 1 previous similar message [ 8233.981698] Lustre: lustre-MDT0001: new disk, initializing [ 8234.155430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8234.169315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8238.453647] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8247.696875] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8253.271934] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8260.412645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8260.783809] Lustre: lustre-OST0000: new disk, initializing [ 8260.787456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8260.794949] Lustre: 267141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8262.447468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8262.460492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8262.593101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8267.423089] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8276.290612] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8283.117405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.122813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8283.125228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8283.125239] Lustre: Skipped 7 previous similar messages [ 8283.133580] Lustre: Skipped 6 previous similar messages [ 8283.157973] LustreError: Skipped 2 previous similar messages [ 8288.062797] Lustre: server umount lustre-OST0000 complete [ 8298.003859] Lustre: server umount lustre-MDT0000 complete [ 8298.467429] LustreError: 265183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.483726] LustreError: 265183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 8301.879216] LustreError: 265978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948583 with bad export cookie 11772373398836895709 [ 8301.894475] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8302.276499] Lustre: server umount lustre-MDT0001 complete [ 8313.381278] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8316.225636] Key type lgssc unregistered [ 8316.587982] LNet: 268899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8316.594213] LNetError: 268899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8316.607989] LNet: Removed LNI 192.168.204.121@tcp [ 8317.490245] Key type .llcrypt unregistered [ 8317.496898] Key type ._llcrypt unregistered [ 8343.417695] Key type ._llcrypt registered [ 8343.420087] Key type .llcrypt registered [ 8343.539621] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8344.730893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8344.765787] alg: No test for adler32 (adler32-zlib) [ 8345.854777] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8346.087224] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8347.792270] Key type lgssc registered [ 8349.236745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8356.836756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8356.852936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.392870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.180602] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8368.615343] LustreError: 270360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.634195] LustreError: 270360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8371.191819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8371.573941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8375.849539] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8383.808538] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8388.609876] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8394.259710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.584960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.925748] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8403.894734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8408.093675] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8409.226078] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8412.136540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8412.147732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8412.173126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.178991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8414.183080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8414.188722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.228585] Lustre: Skipped 1 previous similar message [ 8417.612421] Lustre: server umount lustre-OST0000 complete [ 8422.881356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8422.890415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.909604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8427.562074] Lustre: server umount lustre-MDT0000 complete [ 8429.543546] LustreError: 270360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.561484] LustreError: 270360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8431.134994] LustreError: 270339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948713 with bad export cookie 7063393907076199501 [ 8431.138534] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8431.143704] LustreError: 270339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8431.407472] Lustre: server umount lustre-MDT0001 complete [ 8441.342930] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8444.668922] Key type lgssc unregistered [ 8444.998948] LNet: 273947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8445.007336] LNetError: 273947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8445.020040] LNet: Removed LNI 192.168.204.121@tcp [ 8446.091378] Key type .llcrypt unregistered [ 8446.093374] Key type ._llcrypt unregistered [ 8464.347520] Key type ._llcrypt registered [ 8464.349479] Key type .llcrypt registered [ 8464.453914] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8465.310049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8465.333401] alg: No test for adler32 (adler32-zlib) [ 8466.417172] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8466.618126] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8468.257115] Key type lgssc registered [ 8469.221939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8479.693442] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8479.733390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.392478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8500.513406] LustreError: 275139:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 8527.200247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8532.692697] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8539.627640] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8541.458665] Lustre: server umount lustre-OST0000 complete [ 8557.984308] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8560.803972] Key type lgssc unregistered [ 8561.176792] LNet: 276769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8561.188426] LNetError: 276769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8561.209556] LNet: Removed LNI 192.168.204.121@tcp [ 8561.992759] Key type .llcrypt unregistered [ 8562.001412] Key type ._llcrypt unregistered [ 8603.375304] Key type ._llcrypt registered [ 8603.376911] Key type .llcrypt registered [ 8603.505605] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8604.515344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8604.584142] alg: No test for adler32 (adler32-zlib) [ 8605.845500] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 8606.083531] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8607.752735] Key type lgssc registered [ 8609.124093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8622.547211] loop0: detected capacity change from 0 to 204800000 [ 8622.709614] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8622.728350] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8622.990131] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8623.945746] loop0: detected capacity change from 0 to 204800000 [ 8624.122963] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8624.132680] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8624.496883] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8625.414390] loop0: detected capacity change from 0 to 204800000 [ 8625.704216] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8625.717447] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8625.916332] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8626.990278] loop0: detected capacity change from 0 to 204800000 [ 8627.158985] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8628.176267] loop0: detected capacity change from 0 to 204800000 [ 8628.213307] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8632.050422] loop0: detected capacity change from 0 to 204800000 [ 8632.115055] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8636.452133] loop0: detected capacity change from 0 to 204800000 [ 8636.483400] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8640.516343] loop0: detected capacity change from 0 to 204800000 [ 8640.549987] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8643.918629] loop0: detected capacity change from 0 to 204800000 [ 8644.006822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8644.029571] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.076910] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8644.083403] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8644.091498] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8644.099387] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8644.106959] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8644.113701] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8644.119908] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8644.127358] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8644.135387] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8644.145097] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8644.155578] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8644.164416] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8644.173666] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8644.186166] Lustre: 280177:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8649.385359] Lustre: server umount MGS complete [ 8650.073255] loop0: detected capacity change from 0 to 204800000 [ 8650.130646] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.596969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.646188] Lustre: lustre-MDT0000: Aborting client recovery [ 8651.675414] loop1: detected capacity change from 0 to 204800000 [ 8651.827846] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.905221] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8651.913189] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8651.923723] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8651.938624] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8651.954130] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8651.963345] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8651.970986] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8651.990570] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8652.155502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8652.185993] Lustre: lustre-MDT0001: Aborting client recovery [ 8653.059382] loop2: detected capacity change from 0 to 204800000 [ 8653.276084] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8653.392135] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8653.416330] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8653.428615] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8653.444658] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8653.463064] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8653.482256] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8653.494968] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8653.634931] Lustre: Found index 0 for lustre-OST0000, updating log [ 8653.701074] Lustre: 280708:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8653.835448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8653.858860] Lustre: lustre-OST0000: Aborting recovery [ 8654.826518] loop3: detected capacity change from 0 to 204800000 [ 8654.971392] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.089434] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8655.101474] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8655.115677] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8655.135623] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8655.149186] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8655.168650] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8655.177437] Lustre: 280773:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8655.198090] Lustre: Found index 1 for lustre-OST0001, updating log [ 8655.265142] Lustre: 280779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8655.374048] Lustre: lustre-OST0001: Aborting recovery [ 8660.466694] LustreError: 280554:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8661.497569] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8661.503049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8661.548883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8661.589756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8662.505262] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8662.526814] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8663.342125] LustreError: 280521:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 638us: rc = -2 [ 8663.901724] LustreError: 280522:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 881us: rc = -2 [ 8663.913628] LustreError: 280522:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8677.861105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8677.863654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.878149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8677.897048] Lustre: Skipped 2 previous similar messages [ 8682.625924] Lustre: server umount lustre-MDT0000 complete [ 8682.984317] LustreError: 281012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8683.019201] LustreError: 281012:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8683.668925] LustreError: 280506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948965 with bad export cookie 17629015701273809193 [ 8683.678383] LustreError: 280506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8683.680190] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8688.097773] LustreError: 280518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8688.099504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.108536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8688.108545] Lustre: Skipped 3 previous similar messages [ 8688.125609] LustreError: 280518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8688.136193] Lustre: Skipped 2 previous similar messages [ 8690.312582] Lustre: server umount lustre-MDT0001 complete [ 8697.636775] Lustre: server umount lustre-OST0000 complete [ 8705.068872] Lustre: server umount lustre-OST0001 complete [ 8713.432930] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 06:16:32 (1788948992) [ 8735.460421] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8775.348605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.684717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8775.733832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8775.925600] Lustre: lustre-MDT0000: new disk, initializing [ 8776.105468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8776.114786] Lustre: Skipped 1 previous similar message [ 8776.152490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8780.695168] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8788.030345] Lustre: server umount lustre-MDT0000 complete [ 8805.360458] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 06:18:04 (1788949084) [ 8833.232204] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 8844.223163] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8879.303553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.548816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8879.587135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8879.595990] Lustre: Skipped 1 previous similar message [ 8879.712332] Lustre: lustre-MDT0000: new disk, initializing [ 8879.807742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8879.819359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8883.445681] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8893.134666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.218272] Lustre: 287882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8893.262141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8893.265994] Lustre: Skipped 1 previous similar message [ 8893.329330] Lustre: lustre-MDT0001: new disk, initializing [ 8893.450509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8893.490884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8893.500684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8898.389371] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8908.145179] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8914.086088] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8920.874574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8921.164332] Lustre: lustre-OST0000: new disk, initializing [ 8921.173708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8921.186352] Lustre: 289853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8921.285440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.314792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8927.323890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8927.443543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8927.917504] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8937.661055] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8942.910898] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8953.939437] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8965.950878] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9009.635379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9009.635714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.642401] LustreError: Skipped 1 previous similar message [ 9009.644054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9009.651710] Lustre: Skipped 1 previous similar message [ 9009.666376] Lustre: Skipped 2 previous similar messages [ 9012.657793] Lustre: server umount lustre-OST0000 complete [ 9019.878683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.893308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9019.898282] Lustre: Skipped 1 previous similar message [ 9023.694988] Lustre: server umount lustre-MDT0000 complete [ 9024.995043] LustreError: 287893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.004382] LustreError: 287893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9027.877141] LustreError: 288688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949309 with bad export cookie 17629015701273812147 [ 9027.878751] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.887025] LustreError: 288688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9028.163656] Lustre: server umount lustre-MDT0001 complete [ 9037.744801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.695658] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9047.749340] Lustre: server umount MGS complete [ 9061.356335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.932974] LustreError: 293859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.951508] LustreError: 293872:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9062.021033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9067.394623] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9067.490095] LustreError: 293860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.510131] LustreError: 293860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9076.707080] LustreError: 293860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.723750] LustreError: 293860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9076.942610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.675777] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9091.348469] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9096.294774] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9103.341129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9110.140372] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9117.280919] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.879071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.887285] Lustre: Skipped 1 previous similar message [ 9147.892415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9147.896476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9147.899611] LustreError: Skipped 1 previous similar message [ 9147.909319] Lustre: Skipped 1 previous similar message [ 9150.665681] Lustre: server umount lustre-OST0000 complete [ 9154.534582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9158.115287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9158.117297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.125203] Lustre: Skipped 1 previous similar message [ 9158.133431] Lustre: Skipped 2 previous similar messages [ 9160.497600] Lustre: server umount lustre-MDT0000 complete [ 9163.233922] LustreError: 293854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9163.244726] LustreError: 293854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9163.262447] LustreError: 293839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949445 with bad export cookie 17629015701273813995 [ 9163.264772] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9163.272269] LustreError: 293839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9163.482285] Lustre: server umount lustre-MDT0001 complete [ 9175.288968] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 9179.437023] Key type lgssc unregistered [ 9179.969879] LNet: 297805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9179.977880] LNetError: 297805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9180.032194] LNet: Removed LNI 192.168.204.121@tcp [ 9181.444759] Key type .llcrypt unregistered [ 9181.455696] Key type ._llcrypt unregistered [ 9225.285058] Key type ._llcrypt registered [ 9225.287166] Key type .llcrypt registered [ 9225.486733] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9241.859531] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9243.387562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9243.480223] alg: No test for adler32 (adler32-zlib) [ 9244.680987] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 9244.958256] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 9246.664441] Key type lgssc registered [ 9247.827153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9283.937593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9283.973379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9285.188058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9285.239318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9285.352974] Lustre: lustre-MDT0000: new disk, initializing [ 9285.424750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.459319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9289.472563] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9301.596230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.753108] Lustre: 301777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9301.866255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9301.876885] Lustre: Skipped 1 previous similar message [ 9302.073507] Lustre: lustre-MDT0001: new disk, initializing [ 9302.235296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9302.305166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9302.337028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9308.046477] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9319.234192] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9326.364440] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9333.723476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9334.052662] Lustre: lustre-OST0000: new disk, initializing [ 9334.071941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9334.089726] Lustre: 303745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9334.227939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.527613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9341.545515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9341.730962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9341.912885] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9351.677884] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9358.990406] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9370.181853] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9376.250484] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9418.721926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9418.734620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.737994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9418.746634] Lustre: Skipped 1 previous similar message [ 9418.758849] Lustre: Skipped 1 previous similar message [ 9423.848128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.855835] Lustre: Skipped 1 previous similar message [ 9428.962736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9428.979102] Lustre: Skipped 1 previous similar message [ 9433.056134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9433.275515] Lustre: server umount lustre-OST0000 complete [ 9439.206032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.218613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.222169] Lustre: Skipped 1 previous similar message [ 9443.175569] Lustre: server umount lustre-MDT0000 complete [ 9446.741538] LustreError: 304602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949728 with bad export cookie 12126843196251553926 [ 9446.747671] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9446.756280] LustreError: 304602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9447.103718] Lustre: server umount lustre-MDT0001 complete [ 9456.019811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9459.798405] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9464.572531] Lustre: server umount MGS complete [ 9476.163455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.776448] LustreError: 307612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.793534] LustreError: 307612:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9476.808507] LustreError: 307625:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9476.894373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9481.036815] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9482.218935] LustreError: 307613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9486.306150] LustreError: 307612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9489.831047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.676278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9496.657914] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9506.338873] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9512.827341] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9521.001088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9521.370528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9529.057213] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9536.814861] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9561.571622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9561.577635] LustreError: Skipped 1 previous similar message [ 9561.581965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.582952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9561.599570] Lustre: Skipped 2 previous similar messages [ 9566.560959] Lustre: server umount lustre-OST0000 complete [ 9571.816711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9571.818786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9571.838985] Lustre: Skipped 1 previous similar message [ 9571.843912] Lustre: Skipped 2 previous similar messages [ 9576.359142] Lustre: server umount lustre-MDT0000 complete [ 9576.929502] LustreError: 307608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9576.945119] LustreError: 307608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9580.048041] LustreError: 307593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949861 with bad export cookie 12126843196251555753 [ 9580.052336] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9580.066047] LustreError: 307593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9580.470077] Lustre: server umount lustre-MDT0001 complete [ 9591.898914] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 9594.381401] Key type lgssc unregistered [ 9594.661237] LNet: 311556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9594.669737] LNetError: 311556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9594.695599] LNet: Removed LNI 192.168.204.121@tcp [ 9595.824974] Key type .llcrypt unregistered [ 9595.829386] Key type ._llcrypt unregistered [ 9619.013617] Key type ._llcrypt registered [ 9619.015480] Key type .llcrypt registered [ 9619.150818] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9632.161492] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9633.197540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9633.224450] alg: No test for adler32 (adler32-zlib) [ 9634.459412] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [ 9634.704628] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 9636.448627] Key type lgssc registered [ 9637.646673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9666.349317] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9685.398661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9685.414328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.847475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9686.925466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9687.093453] Lustre: lustre-MDT0000: new disk, initializing [ 9687.194902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9687.226959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9692.622303] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9704.568172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9704.877026] Lustre: lustre-OST0000: new disk, initializing [ 9704.884785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9704.898312] Lustre: Skipped 1 previous similar message [ 9704.907197] Lustre: 315586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9705.114315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9713.209501] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9714.340426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9714.357213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9714.419520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10605.036842] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.063610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10605.469217] Lustre: server umount lustre-MDT0000 complete [10608.886985] LustreError: 314773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950890 with bad export cookie 4174618916723684029 [10608.903900] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10608.915317] LustreError: 314773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10611.118528] Lustre: server umount lustre-OST0000 complete [10634.931497] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [10646.290333] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10680.900189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.152725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10681.184255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10681.282733] Lustre: lustre-MDT0000: new disk, initializing [10681.382307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10681.403766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10686.189206] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10697.998641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.188579] Lustre: 319932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10698.239069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10698.242582] Lustre: Skipped 1 previous similar message [10698.348296] Lustre: lustre-MDT0001: new disk, initializing [10698.461058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10698.505334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10698.518358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10703.080269] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10713.300292] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.150771] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10725.984361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.233300] Lustre: lustre-OST0000: new disk, initializing [10726.242838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10726.262723] Lustre: 321899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10726.390598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10732.725389] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10734.665239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10734.672042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10734.805173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10741.718839] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10746.283536] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10746.576269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10751.914610] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10752.134871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10755.554157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.557394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10755.572442] Lustre: Skipped 1 previous similar message [10755.582204] Lustre: Skipped 1 previous similar message [10760.676865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10760.689823] Lustre: Skipped 1 previous similar message [10765.798125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10765.815172] Lustre: Skipped 1 previous similar message [10768.353671] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10768.528358] Lustre: server umount lustre-OST0000 complete [10775.522060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10775.529846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.547535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10778.512392] Lustre: server umount lustre-MDT0000 complete [10782.191465] LustreError: 319924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951064 with bad export cookie 4174618916726624841 [10782.194476] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10782.208239] LustreError: 319924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10782.537833] Lustre: server umount lustre-MDT0001 complete [10785.362870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10786.806745] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 06:51:06 (1788951066) [10793.046410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.574300] LustreError: 324216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.599112] LustreError: 324216:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10793.646773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10799.080433] LustreError: 324217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10799.111380] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10803.174819] LustreError: 324216:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.172987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.294682] LustreError: 324217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.654612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10813.158391] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10820.344696] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10826.322590] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10833.582267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10834.116188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10841.208674] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10849.165411] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10856.801983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.018949] Lustre: lustre-OST0001: new disk, initializing [10857.023443] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10857.036792] Lustre: 327053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10857.144064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10863.683102] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10866.783684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10866.805758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10866.856991] LustreError: 327519:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10867.877692] LustreError: 327519:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10867.889154] LustreError: 327519:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10868.963286] LustreError: 327519:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10868.977889] LustreError: 327519:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10871.011927] LustreError: 327529:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10871.019362] LustreError: 327529:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10874.296781] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10874.595535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10875.109389] LustreError: 327529:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10875.120103] LustreError: 327529:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10880.487882] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f0cd0a3d800.ost_server_uuid 50 [10882.110975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f0cd0a3d800.ost_server_uuid in FULL state after 0 sec [10883.395268] LustreError: 327529:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10883.403918] LustreError: 327529:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10888.413672] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0cd0a3d800.ost_server_uuid 50 [10889.873715] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f0cd0a3d800.ost_server_uuid in FULL state after 0 sec [10891.642330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10907.112259] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10907.112707] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10907.126621] LustreError: Skipped 1 previous similar message [10907.132497] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10907.137750] Lustre: Skipped 2 previous similar messages [10907.152276] Lustre: Skipped 2 previous similar messages [10911.440783] Lustre: server umount lustre-OST0001 complete [10913.255058] LustreError: 327520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.271772] LustreError: 327520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10918.370982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.383047] Lustre: Skipped 1 previous similar message [10923.529513] Lustre: server umount lustre-OST0000 complete [10928.610766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.622703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10928.633645] Lustre: Skipped 7 previous similar messages [10933.558979] Lustre: server umount lustre-MDT0000 complete [10933.731794] LustreError: 324217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.746067] LustreError: 324217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10937.290911] LustreError: 324198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951219 with bad export cookie 4174618916726625723 [10937.293854] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10937.307359] LustreError: 324198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10937.708336] Lustre: server umount lustre-MDT0001 complete [10949.854683] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [10952.945440] Key type lgssc unregistered [10953.235289] LNet: 329742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10953.246262] LNetError: 329742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10953.263481] LNet: Removed LNI 192.168.204.121@tcp [10954.183450] Key type .llcrypt unregistered [10954.187656] Key type ._llcrypt unregistered [10979.178222] Key type ._llcrypt registered [10979.185472] Key type .llcrypt registered [10979.316143] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10980.263474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10980.276712] alg: No test for adler32 (adler32-zlib) [10981.403285] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [10981.653723] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [10983.368215] Key type lgssc registered [10984.592990] Lustre: Echo OBD driver; http://www.lustre.org/ [10993.345897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10993.387630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10994.916383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11000.326640] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11004.395908] LustreError: 331159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.423988] LustreError: 331159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11008.182546] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11009.518761] LustreError: 331158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.634642] LustreError: 331159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.945182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11015.494658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11020.234894] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11029.090858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11029.425850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11035.574901] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11039.670334] LustreError: 332841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.685392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11042.954759] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11045.860343] LustreError: 333375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.878994] LustreError: 333375:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11048.755346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11048.983440] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11054.064811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11054.221802] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11061.446141] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11068.248195] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11068.478804] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11171.816063] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11171.820836] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11171.831083] LustreError: Skipped 1 previous similar message [11171.836130] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11171.865409] Lustre: Skipped 1 previous similar message [11176.935637] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11176.948972] Lustre: Skipped 2 previous similar messages [11177.291429] Lustre: server umount lustre-OST0001 complete [11182.055941] LustreError: 332841:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.058867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11182.069525] LustreError: 332841:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11182.071444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11182.086738] Lustre: Skipped 1 previous similar message [11187.174246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11187.196174] Lustre: Skipped 2 previous similar messages [11187.922558] Lustre: server umount lustre-OST0000 complete [11192.297315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11192.302466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11192.321517] Lustre: Skipped 1 previous similar message [11192.332536] Lustre: Skipped 1 previous similar message [11198.594591] Lustre: server umount lustre-MDT0000 complete [11202.411397] LustreError: 331141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951484 with bad export cookie 10841100970940654748 [11202.430981] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11202.433270] LustreError: 331141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11202.880811] Lustre: server umount lustre-MDT0001 complete [11212.497507] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [11215.283227] Key type lgssc unregistered [11215.558421] LNet: 336098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11215.570118] LNetError: 336098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11215.582267] LNet: Removed LNI 192.168.204.121@tcp [11216.277473] Key type .llcrypt unregistered [11216.280430] Key type ._llcrypt unregistered [11262.653653] Key type ._llcrypt registered [11262.656937] Key type .llcrypt registered [11262.809979] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11264.478340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11264.503232] alg: No test for adler32 (adler32-zlib) [11265.604549] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [11265.829839] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [11267.512307] Key type lgssc registered [11268.598252] Lustre: Echo OBD driver; http://www.lustre.org/ [11277.503812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11277.533952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.259141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11283.199486] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11289.584899] LustreError: 338386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11289.603824] LustreError: 338386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11291.453279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.869982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11296.456941] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11299.380796] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11306.789175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11307.199702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11315.296754] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11316.411052] LustreError: 339880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.829636] LustreError: 339878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11324.925173] LustreError: 339880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11324.949802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11324.955271] LustreError: 339880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11325.970175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11326.297494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11331.583835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11333.116649] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11342.377131] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11353.176588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11363.810118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11363.817632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11363.830551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.396954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.406465] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11367.424858] Lustre: Skipped 2 previous similar messages [11369.353420] Lustre: server umount lustre-MDT0000 complete [11372.526149] LustreError: 338381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.569337] LustreError: 338381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11373.263869] LustreError: 339126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951655 with bad export cookie 8551436944247242164 [11373.280187] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11373.283616] LustreError: 339126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11373.682628] Lustre: server umount lustre-MDT0001 complete [11387.425449] Lustre: server umount lustre-OST0000 complete [11401.787237] Lustre: server umount lustre-OST0001 complete [11418.598686] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11429.114539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.670612] LustreError: 343665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.810540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11434.593975] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11444.022047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.569194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11449.251828] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11452.014970] Lustre: 344808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11458.407484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.273906] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11469.095669] LustreError: 345159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.136592] LustreError: 345159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11474.325826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11474.633619] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11474.639940] Lustre: Skipped 1 previous similar message [11475.656496] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11475.709663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11480.609776] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11487.115208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11490.317277] Lustre: 346679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11495.910228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11495.931823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.960846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11495.972064] Lustre: Skipped 3 previous similar messages [11501.022346] Lustre: server umount lustre-MDT0000 complete [11501.545262] LustreError: 343665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.560807] LustreError: 343665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11504.277110] LustreError: 343646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951786 with bad export cookie 8551436944247243865 [11504.280503] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11504.300520] LustreError: 343646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11504.657096] Lustre: server umount lustre-MDT0001 complete [11518.958868] Lustre: server umount lustre-OST0000 complete [11532.895555] Lustre: server umount lustre-OST0001 complete [11536.643951] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11538.184626] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11539.889631] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 07:03:39 (1788951819) [11561.049602] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11569.209711] loop0: detected capacity change from 0 to 1099511627776 [11569.923545] SGI XFS with ACLs, security attributes, quota, no debug enabled [11569.929896] *** VALIDATE xfs *** [11569.946442] XFS (loop0): Mounting V5 Filesystem [11569.978445] XFS (loop0): Ending clean mount [11571.118851] loop1: detected capacity change from 0 to 18691697672192 [11661.675530] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [11673.052926] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11709.907416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.135994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11710.175596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11710.273593] Lustre: lustre-MDT0000: new disk, initializing [11710.353029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11710.371657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11715.116466] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11726.182608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.362224] Lustre: 351828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11726.456683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11726.467818] Lustre: Skipped 1 previous similar message [11726.618817] Lustre: lustre-MDT0001: new disk, initializing [11726.811267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11726.827546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11731.135945] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11740.629273] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11747.021905] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11754.308685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11754.628928] Lustre: lustre-OST0000: new disk, initializing [11754.633228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11754.648227] Lustre: 353796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11755.809605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11755.826353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11756.044063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11761.226751] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11771.269594] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11776.594550] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11776.861104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11782.238820] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11782.498278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11786.729870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11786.730099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.743446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11786.743455] Lustre: Skipped 4 previous similar messages [11786.775168] Lustre: Skipped 4 previous similar messages [11790.769377] Lustre: server umount lustre-OST0000 complete [11796.973634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11796.982135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11797.001672] Lustre: Skipped 1 previous similar message [11797.022579] Lustre: Skipped 2 previous similar messages [11806.179170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.196139] Lustre: Skipped 3 previous similar messages [11809.248613] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11809.503570] Lustre: server umount lustre-MDT0000 complete [11811.297968] LustreError: 354142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.312298] LustreError: 354142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11813.780376] LustreError: 351820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788952095 with bad export cookie 8551436944247245419 [11813.786449] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.793529] LustreError: 351820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11814.257921] Lustre: server umount lustre-MDT0001 complete [11823.445517] XFS (loop0): Unmounting Filesystem [11828.684445] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 07:08:27 (1788952107) [11836.105037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.780847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.791482] Lustre: Skipped 2 previous similar messages [11841.595873] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11850.755157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.096434] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11864.608575] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11871.357956] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11879.009814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11886.325111] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11893.788692] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11907.042846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11907.052908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11907.058934] LustreError: Skipped 2 previous similar messages [11907.070421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11907.078949] Lustre: Skipped 1 previous similar message [11907.107459] Lustre: Skipped 1 previous similar message [11911.838872] Lustre: server umount lustre-OST0000 complete [11922.007794] Lustre: server umount lustre-MDT0000 complete [11925.392139] LustreError: 356335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788952207 with bad export cookie 8551436944247246301 [11925.398995] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11925.402755] LustreError: 356335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11925.802920] Lustre: server umount lustre-MDT0001 complete [11937.169652] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [11940.230545] Key type lgssc unregistered [11940.583834] LNet: 359984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11940.594442] LNetError: 359984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11941.681117] LNet: Removed LNI 192.168.204.121@tcp [11942.755197] Key type .llcrypt unregistered [11942.757114] Key type ._llcrypt unregistered [11971.565545] Key type ._llcrypt registered [11971.566869] Key type .llcrypt registered [11971.681368] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11973.197640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11973.249802] alg: No test for adler32 (adler32-zlib) [11974.524635] Lustre: Lustre: Build Version: 2.17.58_40_gec2f674 [11975.062025] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [11976.801122] Key type lgssc registered [11978.027404] Lustre: Echo OBD driver; http://www.lustre.org/ [11986.908118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11986.922264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11988.327294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11994.028246] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11998.697029] LustreError: 361550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.719323] LustreError: 361550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12002.688350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12003.046120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12007.757092] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12016.047815] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12022.248386] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12029.543758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12030.065660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12036.169955] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12044.179307] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12057.385491] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [12058.093055] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12058.119803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12058.130834] Lustre: Skipped 1 previous similar message [12061.429863] Lustre: server umount lustre-MDT0001 complete [12062.483131] LustreError: 361545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.605272] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.620668] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12070.706887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.799955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12070.808766] Lustre: Found index 1 for lustre-MDT0001, updating log [12070.843898] Lustre: 361536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12071.105881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12075.348792] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12076.527358] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12079.614225] LustreError: 365149:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12079.627848] LustreError: 365149:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12079.638216] LustreError: 361542:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12079.688988] LustreError: 365152:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -17 [12079.696572] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12079.696572] [12383.713699] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12383.718727] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12383.723936] Lustre: Skipped 1 previous similar message [12383.728898] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12383.746170] Lustre: Skipped 1 previous similar message [12386.830651] Lustre: server umount lustre-MDT0001 complete [12388.837013] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12388.844496] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12393.956275] LustreError: 362277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12394.009788] LustreError: 362277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12399.949226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12400.111661] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12400.116623] Lustre: Found index 1 for lustre-MDT0001, updating log [12400.161505] Lustre: 361539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12400.425973] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (not set up) [12400.449852] Lustre: Skipped 1 previous similar message [12400.826450] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12406.254312] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12407.638153] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12408.328313] LustreError: 373743:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12408.335198] LustreError: 373743:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12408.353817] LustreError: 361542:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12408.363613] LustreError: 361542:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12408.412392] LustreError: 373750:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12408.417390] LustreError: 373750:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12408.421160] LustreError: 373750:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -17 [12408.425961] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.121@tcp [12408.425961] [12717.880971] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [12718.566810] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12718.567889] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12718.580490] Lustre: Skipped 1 previous similar message [12728.091102] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [12728.103913] Lustre: Skipped 5 previous similar messages [12728.289644] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12728.459057] Lustre: server umount lustre-MDT0001 complete [12728.803956] LustreError: 362769:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.824312] LustreError: 362769:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12738.016197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12738.081610] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12738.084891] Lustre: Found index 1 for lustre-MDT0001, updating log [12738.111325] Lustre: 361536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12738.489247] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12742.179885] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12743.705262] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12744.201575] LustreError: 382611:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12744.212027] LustreError: 382611:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12744.218565] LustreError: 361542:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12744.221942] LustreError: 361542:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12744.254252] LustreError: 382619:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12744.261517] LustreError: 382619:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -114 [12744.267631] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12744.267631] [13066.212304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13066.224944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13066.231231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13066.241073] Lustre: Skipped 3 previous similar messages [13066.248107] Lustre: Skipped 3 previous similar messages [13070.722564] Lustre: server umount lustre-MDT0000 complete [13071.337112] LustreError: 374098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.347712] LustreError: 374098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13074.553592] LustreError: 361530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953356 with bad export cookie 5310582243619967690 [13074.567477] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13074.568228] LustreError: 361530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13076.471844] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.068662] Lustre: server umount lustre-MDT0001 complete [13085.386622] Lustre: server umount lustre-OST0000 complete [13095.799223] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [13107.309580] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13147.462827] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 07:30:26 (1788953426) [13154.259925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13154.548705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13154.591288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13154.685431] Lustre: lustre-MDT0000: new disk, initializing [13154.793105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13154.815575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13160.003921] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13169.386947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.594492] Lustre: 393430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13169.683915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13169.696903] Lustre: Skipped 1 previous similar message [13169.832590] Lustre: lustre-MDT0001: new disk, initializing [13170.018435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13170.053604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13170.063815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13177.623508] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13192.292655] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13199.179777] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13206.059758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13206.380371] Lustre: lustre-OST0000: new disk, initializing [13206.389486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13206.402807] Lustre: 395400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13206.516448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13214.452836] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13216.350528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13216.356587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13216.575251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13226.014100] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13235.177859] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13235.187239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13235.201881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13235.209709] Lustre: Skipped 2 previous similar messages [13236.887267] Lustre: server umount lustre-MDT0000 complete [13237.224142] LustreError: 394305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13237.263095] LustreError: 394305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13241.498309] LustreError: 394235:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13241.500803] LustreError: 393421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953523 with bad export cookie 5310582243619969643 [13241.501904] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13241.569682] LustreError: 393421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13241.970765] Lustre: server umount lustre-MDT0001 complete [13257.104146] Lustre: server umount lustre-OST0000 complete [13289.862235] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [13301.859215] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13333.636967] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13049 sec ========= 07:33:31 (1788953611) [13336.390982] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:33:34 (1788953614) === [13342.222359] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:33:40 (1788953620) === [13387.941344] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13392.254688] Key type lgssc unregistered [13392.752744] LNet: 401767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13392.775672] LNetError: 401767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13392.796211] LNet: Removed LNI 192.168.204.121@tcp [13394.217185] Key type .llcrypt unregistered [13394.231827] Key type ._llcrypt unregistered