[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 528176745 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003070] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006011] kvm-guest: setup PV IPIs [ 0.009734] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.013020] LSM: Security Framework initializing [ 0.014056] Yama: becoming mindful. [ 0.015029] SELinux: Initializing. [ 0.016068] *** VALIDATE selinux *** [ 0.025474] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030336] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034133] *** VALIDATE tmpfs *** [ 0.035487] *** VALIDATE proc *** [ 0.036271] *** VALIDATE cgroup *** [ 0.037014] *** VALIDATE cgroup2 *** [ 0.039212] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042034] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.046516] debug: unmapping init [mem 0xffffffffaa859000-0xffffffffaa860fff] [ 0.048179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049749] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050026] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052013] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054018] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057331] rcu: Hierarchical SRCU implementation. [ 0.059412] smp: Bringing up secondary CPUs ... [ 0.060605] x86: Booting SMP configuration: [ 0.061034] .... node #0, CPUs: #1 #2 #3 [ 0.069669] smp: Brought up 1 node, 4 CPUs [ 0.071014] smpboot: Max logical packages: 1 [ 0.072021] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.152022] node 0 deferred pages initialised in 77ms [ 0.155141] devtmpfs: initialized [ 0.157318] x86/mm: Memory block size: 128MB [ 0.160979] gcov: version magic: 0x41383552 [ 0.164337] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.169188] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.172390] pinctrl core: initialized pinctrl subsystem [ 0.174215] [ 0.174729] ************************************************************* [ 0.177012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180017] ** ** [ 0.183015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.186015] ** ** [ 0.188012] ** This means that this kernel is built to expose internal ** [ 0.191018] ** IOMMU data structures, which may compromise security on ** [ 0.193012] ** your system. ** [ 0.196015] ** ** [ 0.199021] ** If you see this message and you are not debugging the ** [ 0.201012] ** kernel, report this immediately to your vendor! ** [ 0.204012] ** ** [ 0.206013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209015] ************************************************************* [ 0.211875] NET: Registered protocol family 16 [ 0.214494] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225012] cpuidle: using governor menu [ 0.226858] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227586] PCI: Using configuration type 1 for base access [ 0.228193] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243032] cryptd: max_cpu_qlen set to 1000 [ 0.248204] ACPI: Added _OSI(Module Device) [ 0.250025] ACPI: Added _OSI(Processor Device) [ 0.252023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.253023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.258180] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.266525] ACPI: Interpreter enabled [ 0.268065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.269011] ACPI: Using IOAPIC for interrupt routing [ 0.270112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.273340] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.283014] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.292123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299919] acpiphp: Slot [2] registered [ 0.302238] acpiphp: Slot [5] registered [ 0.303251] acpiphp: Slot [6] registered [ 0.305263] acpiphp: Slot [7] registered [ 0.307228] acpiphp: Slot [8] registered [ 0.310079] acpiphp: Slot [9] registered [ 0.311146] acpiphp: Slot [10] registered [ 0.313186] acpiphp: Slot [3] registered [ 0.315134] acpiphp: Slot [4] registered [ 0.317180] acpiphp: Slot [11] registered [ 0.318145] acpiphp: Slot [12] registered [ 0.320109] acpiphp: Slot [13] registered [ 0.322123] acpiphp: Slot [14] registered [ 0.323095] acpiphp: Slot [15] registered [ 0.325110] acpiphp: Slot [16] registered [ 0.326093] acpiphp: Slot [17] registered [ 0.327152] acpiphp: Slot [18] registered [ 0.329136] acpiphp: Slot [19] registered [ 0.331118] acpiphp: Slot [20] registered [ 0.332169] acpiphp: Slot [21] registered [ 0.334111] acpiphp: Slot [22] registered [ 0.335105] acpiphp: Slot [23] registered [ 0.337098] acpiphp: Slot [24] registered [ 0.338155] acpiphp: Slot [25] registered [ 0.340114] acpiphp: Slot [26] registered [ 0.341103] acpiphp: Slot [27] registered [ 0.343190] acpiphp: Slot [28] registered [ 0.345183] acpiphp: Slot [29] registered [ 0.347144] acpiphp: Slot [30] registered [ 0.348145] acpiphp: Slot [31] registered [ 0.350093] PCI host bridge to bus 0000:00 [ 0.352039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.355043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.358037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361052] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370260] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373314] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.377630] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.388022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393653] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.398018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403687] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406903] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413085] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.432025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.438028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.444380] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485934] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.496016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.505015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.535536] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.543017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.548020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565037] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.574874] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.590022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.611024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.623140] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.637020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.652020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662493] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.682026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.702029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.722565] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.729426] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.732490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735284] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740042] iommu: Default domain type: Passthrough [ 0.742559] SCSI subsystem initialized [ 0.744188] ACPI: bus type USB registered [ 0.746202] usbcore: registered new interface driver usbfs [ 0.748146] usbcore: registered new interface driver hub [ 0.751126] usbcore: registered new device driver usb [ 0.752270] pps_core: LinuxPPS API ver. 1 registered [ 0.755023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.760096] PTP clock support registered [ 0.763059] EDAC MC: Ver: 3.0.0 [ 0.765158] PCI: Using ACPI for IRQ routing [ 0.767831] NetLabel: Initializing [ 0.768007] NetLabel: domain hash size = 128 [ 0.769012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.771082] NetLabel: unlabeled traffic allowed by default [ 0.773096] vgaarb: loaded [ 0.774320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784023] clocksource: Switched to clocksource kvm-clock [ 0.892463] VFS: Disk quotas dquot_6.6.0 [ 0.893960] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.897126] *** VALIDATE ramfs *** [ 0.898630] *** VALIDATE hugetlbfs *** [ 0.900540] pnp: PnP ACPI init [ 0.903887] pnp: PnP ACPI: found 6 devices [ 0.921231] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.924011] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.926271] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.927786] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.929530] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.931414] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.934176] NET: Registered protocol family 2 [ 0.936720] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.941740] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.945799] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951918] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955780] TCP: Hash tables configured (established 65536 bind 65536) [ 0.958852] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.962162] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.965129] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967415] NET: Registered protocol family 1 [ 0.969671] RPC: Registered named UNIX socket transport module. [ 0.971721] RPC: Registered udp transport module. [ 0.973366] RPC: Registered tcp transport module. [ 0.974920] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.976725] NET: Registered protocol family 44 [ 0.978302] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.980193] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.981727] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.983331] PCI: CLS 0 bytes, default 64 [ 0.984382] Unpacking initramfs... [ 2.410870] debug: unmapping init [mem 0xffff8ccafcc54000-0xffff8ccafffbffff] [ 2.414134] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.415702] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.417741] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.908787] Initialise system trusted keyrings [ 2.910491] Key type blacklist registered [ 2.912545] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.920761] zbud: loaded [ 2.923127] *** VALIDATE nfs *** [ 2.924026] *** VALIDATE nfs4 *** [ 2.924973] pstore: using deflate compression [ 2.927536] Platform Keyring initialized [ 3.023185] NET: Registered protocol family 38 [ 3.024836] Key type asymmetric registered [ 3.026386] Asymmetric key parser 'x509' registered [ 3.028381] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.031270] io scheduler mq-deadline registered [ 3.033241] io scheduler kyber registered [ 3.034950] io scheduler bfq registered [ 3.036743] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.039962] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.043175] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.046744] ACPI: Power Button [PWRF] [ 3.053179] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.060210] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.082590] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.095208] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.119075] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.148051] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.176371] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.181442] Non-volatile memory driver v1.3 [ 3.183296] Linux agpgart interface v0.103 [ 3.215981] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.218873] vda: detected capacity change from 0 to 74694656 [ 3.239793] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.242457] vdb: detected capacity change from 0 to 1073741824 [ 3.265158] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.268052] vdc: detected capacity change from 0 to 2621440000 [ 3.290477] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.294129] vdd: detected capacity change from 0 to 2621440000 [ 3.313408] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.316446] vde: detected capacity change from 0 to 4294967296 [ 3.331557] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.334665] vdf: detected capacity change from 0 to 4294967296 [ 3.341393] libphy: Fixed MDIO Bus: probed [ 3.347126] usbcore: registered new interface driver usbserial_generic [ 3.349683] usbserial: USB Serial support registered for generic [ 3.352023] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.356381] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.358180] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.360593] mousedev: PS/2 mouse device common for all mice [ 3.362898] rtc_cmos 00:05: RTC can wake from S4 [ 3.365363] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.368495] rtc_cmos 00:05: registered as rtc0 [ 3.370136] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.373043] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.374328] intel_pstate: CPU model not supported [ 3.379061] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.382289] hid: raw HID events driver (C) Jiri Kosina [ 3.384499] usbcore: registered new interface driver usbhid [ 3.386565] usbhid: USB HID core driver [ 3.388155] drop_monitor: Initializing network drop monitor service [ 3.390534] Initializing XFRM netlink socket [ 3.392469] NET: Registered protocol family 10 [ 3.395741] Segment Routing with IPv6 [ 3.397424] NET: Registered protocol family 17 [ 3.400180] mpls_gso: MPLS GSO support [ 3.406181] RAS: Correctable Errors collector initialized. [ 3.407720] AVX version of gcm_enc/dec engaged. [ 3.408994] AES CTR mode by8 optimization enabled [ 3.483293] sched_clock: Marking stable (3483193968, 0)->(4433501785, -950307817) [ 3.485793] registered taskstats version 1 [ 3.487942] Loading compiled-in X.509 certificates [ 3.489464] zswap: loaded using pool lzo/zbud [ 3.511875] Key type big_key registered [ 3.529883] Key type encrypted registered [ 3.531305] ima: No TPM chip found, activating TPM-bypass! [ 3.533190] ima: Allocated hash algorithm: sha1 [ 3.534621] ima: No architecture policies found [ 3.536299] evm: Initialising EVM extended attributes: [ 3.537701] evm: security.selinux [ 3.538691] evm: security.ima [ 3.539589] evm: security.capability [ 3.540596] evm: HMAC attrs: 0x1 [ 3.542943] rtc_cmos 00:05: setting system clock to 2026-08-15 13:10:26 UTC (1786799426) [ 3.548427] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 3.551112] debug: unmapping init [mem 0xffffffffaa582000-0xffffffffaa858fff] [ 3.559368] Write protecting the kernel read-only data: 28672k [ 3.562391] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 3.564637] debug: unmapping init [mem 0xffffffffa9514000-0xffffffffa95fffff] [ 3.594670] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.601990] systemd[1]: Detected virtualization kvm. [ 3.603649] systemd[1]: Detected architecture x86-64. [ 3.606422] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.633469] systemd[1]: No hostname configured. [ 3.635676] systemd[1]: Set hostname to . [ 3.638180] random: systemd: uninitialized urandom read (16 bytes read) [ 3.641147] systemd[1]: Initializing machine ID from random generator. [ 3.802574] random: systemd: uninitialized urandom read (16 bytes read) [ 3.804910] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.810268] random: systemd: uninitialized urandom read (16 bytes read) [ 3.813326] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.818079] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.502120] device-mapper: uevent: version 1.0.3 [ 4.504131] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.309213] random: fast init done [ 5.329769] virtio_net virtio0 ens2: renamed from eth0 [ 5.377778] scsi host0: ata_piix [ 5.479982] scsi host1: ata_piix [ 5.481866] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.484352] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.085800] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.045809] random: crng init done [ 10.046996] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.495589] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.601786] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.868237] SELinux: Disabled at runtime. [ 11.926871] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.935079] systemd[1]: Detected virtualization kvm. [ 11.937024] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.439688] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.443162] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.448205] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.454345] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.457815] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.465437] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.468589] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ 12.513041] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.930042] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.300376] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.312802] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.419752] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.435553] EDAC sbridge: Ver: 1.1.2 [ 15.074859] Key type dns_resolver registered [ 15.389321] NFS: Registering the id_resolver key type [ 15.392086] Key type id_resolver registered [ 15.393904] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-server login: [ 42.499582] libcfs: loading out-of-tree module taints kernel. [ 42.522042] Key type ._llcrypt registered [ 42.524056] Key type .llcrypt registered [ 42.582911] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 56.877423] hrtimer: interrupt took 8495018 ns [ 58.179158] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 59.631182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.674344] alg: No test for adler32 (adler32-zlib) [ 61.454886] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 62.570660] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 64.351386] Key type lgssc registered [ 66.734285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.491000] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 128.536765] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 142.945813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 143.013862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.309537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.337268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.399872] Lustre: lustre-MDT0000: new disk, initializing [ 144.492787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.531409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 149.143179] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.588776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.705230] Lustre: 6500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.749183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.762425] Lustre: Skipped 1 previous similar message [ 161.866436] Lustre: lustre-MDT0001: new disk, initializing [ 161.996825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.061507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.080176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 167.026886] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.415161] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.241698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.495641] Lustre: lustre-OST0000: new disk, initializing [ 180.500141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.508589] Lustre: 8438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.588131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.429697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.437594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.534382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.914557] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.153363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.379675] Lustre: lustre-OST0001: new disk, initializing [ 202.383253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.388453] Lustre: 9511:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.456611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.146271] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.015154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.022829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.103229] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.795287] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.579321] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.742448] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.794427] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 242.975857] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 246.194428] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 249.189663] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 251.138051] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 09:14:31 EDT 2026 [ 269.199432] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 270.702501] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 288.224911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.229493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.243528] Lustre: Skipped 2 previous similar messages [ 288.263467] Lustre: Skipped 3 previous similar messages [ 293.334167] Lustre: server umount lustre-MDT0000 complete [ 293.352031] LustreError: 6508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.368876] LustreError: 6508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 297.615114] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799720 with bad export cookie 8804504344266976361 [ 297.615227] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.622465] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 298.481534] LustreError: 6507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.490296] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.492127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.511522] LustreError: 6507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 298.552868] Lustre: Skipped 2 previous similar messages [ 302.499414] LustreError: 6508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.499471] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.511203] LustreError: 6508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 302.525336] Lustre: Skipped 2 previous similar messages [ 304.283576] Lustre: server umount lustre-MDT0001 complete [ 308.469379] Lustre: server umount lustre-OST0000 complete [ 312.273613] Lustre: server umount lustre-OST0001 complete [ 318.661890] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 327.534854] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 363.377106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.661842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 363.685817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 363.836387] Lustre: lustre-MDT0000: new disk, initializing [ 363.962094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.013175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.135725] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 378.728055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.910318] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.982165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.985892] Lustre: Skipped 1 previous similar message [ 379.094256] Lustre: lustre-MDT0001: new disk, initializing [ 379.307591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 379.372872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 379.378747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 383.823616] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 393.623772] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.146363] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 405.812293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.085306] Lustre: lustre-OST0000: new disk, initializing [ 406.090988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.097608] Lustre: 17036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.245019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.490261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 407.506453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 407.630079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 411.327897] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 420.908389] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 424.998298] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 425.159538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.226650] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 429.397590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 433.131074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 433.134288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.139586] LustreError: Skipped 1 previous similar message [ 433.143598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.180717] Lustre: Skipped 1 previous similar message [ 437.323717] Lustre: server umount lustre-OST0000 complete [ 440.801577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 440.810417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.835187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.845063] Lustre: Skipped 1 previous similar message [ 446.948540] Lustre: server umount lustre-MDT0000 complete [ 448.490080] LustreError: 15078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.502900] LustreError: 15078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 450.045974] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799873 with bad export cookie 8804504344266977894 [ 450.051542] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.062425] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 450.392865] Lustre: server umount lustre-MDT0001 complete [ 454.958254] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:17:55 (1786799875) [ 460.819158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.316972] LustreError: 19299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.380844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.266170] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 471.523926] LustreError: 19300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.539408] LustreError: 19300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 472.835795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.312529] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 482.662500] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.200417] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.259086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.785486] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 507.190985] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.609212] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 519.149543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 519.164430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.184336] Lustre: Skipped 1 previous similar message [ 519.193952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.201031] Lustre: Skipped 2 previous similar messages [ 524.038348] Lustre: server umount lustre-MDT0000 complete [ 526.817983] LustreError: 19299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.851266] LustreError: 19299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 527.702147] LustreError: 19279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786799950 with bad export cookie 8804504344266978776 [ 527.703106] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.712758] LustreError: 19279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 528.173518] Lustre: server umount lustre-MDT0001 complete [ 543.004606] Lustre: server umount lustre-OST0000 complete [ 566.771713] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 576.526690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.899314] LustreError: 23681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.981679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.990833] Lustre: Skipped 2 previous similar messages [ 581.342784] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 590.083559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.688862] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 598.043739] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 604.829077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.120385] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 622.985645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 623.556896] Lustre: lustre-OST0001: new disk, initializing [ 623.566233] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 623.581476] Lustre: 25996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 625.570238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 625.586730] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 625.762717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 632.951929] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 642.850750] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 647.441396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 661.474644] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.480386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.501485] Lustre: Skipped 4 previous similar messages [ 661.521287] Lustre: Skipped 6 previous similar messages [ 667.648532] Lustre: server umount lustre-MDT0000 complete [ 671.329084] LustreError: 24820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800094 with bad export cookie 8804504344266979623 [ 671.331597] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.339243] LustreError: 24820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 671.716454] Lustre: server umount lustre-MDT0001 complete [ 685.944669] Lustre: server umount lustre-OST0000 complete [ 687.071510] Lustre: 3635:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786800094/real 1786800094] req@ffff8ccb75e9d180 x1873595057023360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786800110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 687.143137] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.167951] Lustre: Skipped 1 previous similar message [ 689.998488] Lustre: server umount lustre-OST0001 complete [ 709.465322] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 721.535823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.067127] LustreError: 29020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.100499] LustreError: 29020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 722.193502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.202583] Lustre: Skipped 3 previous similar messages [ 726.905697] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 736.177957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.589946] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 744.715175] Lustre: 30167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 751.232090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.789775] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 763.896118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 770.564393] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 778.528511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 783.100805] Lustre: 32047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 793.064096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.081549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 793.085687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 793.085699] Lustre: Skipped 4 previous similar messages [ 793.089708] Lustre: Skipped 3 previous similar messages [ 796.535366] Lustre: server umount lustre-MDT0000 complete [ 800.577249] LustreError: 29002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800223 with bad export cookie 8804504344266980918 [ 800.595722] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.598457] LustreError: 29002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 800.892395] Lustre: server umount lustre-MDT0001 complete [ 814.267287] Lustre: server umount lustre-OST0000 complete [ 827.297796] Lustre: server umount lustre-OST0001 complete [ 835.361786] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:24:15 (1786800255) [ 862.921966] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 872.000676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.288114] LustreError: 34852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.310147] LustreError: 34852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 876.196890] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 885.448041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.480400] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 894.065459] Lustre: 36001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 901.754164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.673562] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 918.971691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 927.094237] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 935.198643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 938.436456] Lustre: 37882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 947.680510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 947.685803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.692868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.695490] Lustre: Skipped 3 previous similar messages [ 949.925614] Lustre: server umount lustre-MDT0000 complete [ 954.418610] LustreError: 34833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800377 with bad export cookie 8804504344266982024 [ 954.424073] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.429546] LustreError: 34833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 955.007472] Lustre: server umount lustre-MDT0001 complete [ 969.650178] Lustre: server umount lustre-OST0000 complete [ 983.494740] Lustre: server umount lustre-OST0001 complete [ 991.371944] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:26:52 (1786800412) [ 1007.289167] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1017.413615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.150683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.160555] Lustre: Skipped 7 previous similar messages [ 1022.547699] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1033.097481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.903567] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1043.515494] Lustre: 41177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1049.869939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1057.259279] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1066.489377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.962132] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1080.584662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1084.352190] Lustre: 43055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1085.199708] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1085.216167] Lustre: Skipped 1 previous similar message [ 1090.022181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1090.029245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.039292] Lustre: Skipped 3 previous similar messages [ 1090.048405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.050932] Lustre: Skipped 5 previous similar messages [ 1095.071522] Lustre: server umount lustre-MDT0000 complete [ 1099.249726] LustreError: 40009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800522 with bad export cookie 8804504344266983130 [ 1099.253349] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.257806] LustreError: 40009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1105.650539] Lustre: server umount lustre-MDT0001 complete [ 1114.501320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.319899] LustreError: 3633:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cca41f9aa00 x1873595057177344/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1129.824721] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1129.893958] LustreError: 43929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.927108] LustreError: 43929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [ 1135.024470] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1138.478398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.484587] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1144.304977] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1144.312764] Lustre: Skipped 1 previous similar message [ 1151.328915] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1152.861941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1158.170163] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1159.814378] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1164.780470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.699410] Lustre: server umount lustre-MDT0000 complete [ 1174.184983] LustreError: 41561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800597 with bad export cookie 8804504344266984257 [ 1174.189257] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.202978] LustreError: 41561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1174.605778] Lustre: server umount lustre-MDT0001 complete [ 1188.281615] Lustre: server umount lustre-OST0000 complete [ 1190.175163] Lustre: 3636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786800597/real 1786800597] req@ffff8ccb4706d180 x1873595057212032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786800613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1192.538939] Lustre: server umount lustre-OST0001 complete [ 1202.368171] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:30:22 (1786800622) [ 1219.945800] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1230.680610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.516833] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1247.396341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.911295] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1255.971536] Lustre: 48811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1262.834959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1268.688913] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1277.294383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.236553] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1292.717939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1297.957170] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1315.795470] Lustre: server umount lustre-MDT0000 complete [ 1319.457921] LustreError: 47641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786800742 with bad export cookie 8804504344266985055 [ 1319.464092] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.474753] LustreError: 47641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1320.099847] Lustre: server umount lustre-MDT0001 complete [ 1333.316862] Lustre: server umount lustre-OST0000 complete [ 1346.634494] Lustre: server umount lustre-OST0001 complete [ 1357.976711] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:32:58 (1786800778) [ 1360.616460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1362.342755] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:33:02 (1786800782) [ 1372.110138] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1380.991316] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1404.629952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.932331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1405.113614] Lustre: lustre-MDT0000: new disk, initializing [ 1405.209265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1410.115903] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1419.043708] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1425.964042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.248191] Lustre: lustre-OST0000: new disk, initializing [ 1426.256510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1426.264231] Lustre: Skipped 1 previous similar message [ 1426.273845] Lustre: 54886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1427.768367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1427.784317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1427.803872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1432.560210] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1442.590876] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1445.529155] Lustre: DEBUG MARKER: create test files [ 1479.143575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1479.146567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.166051] Lustre: Skipped 14 previous similar messages [ 1479.178129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.185810] Lustre: Skipped 22 previous similar messages [ 1492.959123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1493.379956] Lustre: server umount lustre-OST0000 complete [ 1497.624512] Lustre: server umount lustre-MDT0000 complete [ 1507.728952] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1511.122644] Key type lgssc unregistered [ 1511.512620] LNet: 56519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.534891] LNetError: 56519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.556912] LNet: Removed LNI 192.168.204.111@tcp [ 1512.640213] Key type .llcrypt unregistered [ 1512.644851] Key type ._llcrypt unregistered [ 1541.777849] Key type ._llcrypt registered [ 1541.779990] Key type .llcrypt registered [ 1541.893460] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1542.712845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1542.735251] alg: No test for adler32 (adler32-zlib) [ 1543.842085] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1544.240422] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1545.943163] Key type lgssc registered [ 1547.345008] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.970327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1558.989531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.536776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.729771] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1573.290624] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1580.133137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1580.445697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.581302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:97 to 0x240000400:129) [ 1585.654746] Lustre: 59236:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1585.660700] Lustre: 59236:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1585.665557] Lustre: 59236:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1585.673909] Lustre: 59236:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1585.684000] Lustre: 59236:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1585.696084] Lustre: 59236:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1585.973769] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1593.322854] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1598.441371] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1604.165819] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1611.234202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.241897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1611.261117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1612.970727] Lustre: server umount lustre-OST0000 complete [ 1617.442693] Lustre: server umount lustre-MDT0000 complete [ 1629.151233] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1631.754996] Key type lgssc unregistered [ 1632.010613] LNet: 60843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.024722] LNetError: 60843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.062815] LNet: Removed LNI 192.168.204.111@tcp [ 1633.898666] Key type .llcrypt unregistered [ 1633.903263] Key type ._llcrypt unregistered [ 1664.889342] Key type ._llcrypt registered [ 1664.892137] Key type .llcrypt registered [ 1665.001489] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1665.933611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1665.964650] alg: No test for adler32 (adler32-zlib) [ 1666.960939] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1667.195252] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1668.879171] Key type lgssc registered [ 1669.774818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1678.051711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1678.071408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.694335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.908827] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1693.190246] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1699.300167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1699.693951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.811427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1704.906766] Lustre: 63681:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1704.912444] Lustre: 63681:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1704.916720] Lustre: 63681:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1704.921261] Lustre: 63681:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1704.924591] Lustre: 63681:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1704.929223] Lustre: 63681:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1706.040944] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1713.498293] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1719.431868] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1761.249700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1761.268308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.290346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.640670] Lustre: server umount lustre-OST0000 complete [ 1768.860380] Lustre: server umount lustre-MDT0000 complete [ 1782.917904] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1786.911429] Key type lgssc unregistered [ 1787.255474] LNet: 65262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.264543] LNetError: 65262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.275291] LNet: Removed LNI 192.168.204.111@tcp [ 1788.143032] Key type .llcrypt unregistered [ 1788.146320] Key type ._llcrypt unregistered [ 1810.106709] Key type ._llcrypt registered [ 1810.109259] Key type .llcrypt registered [ 1810.205242] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1825.417462] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1826.506018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.523557] alg: No test for adler32 (adler32-zlib) [ 1827.566305] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 1827.838565] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1829.551791] Key type lgssc registered [ 1830.931799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1871.011322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1871.040448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.292361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1872.321569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1872.506025] Lustre: lustre-MDT0000: new disk, initializing [ 1872.608318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.639353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1877.408309] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1889.325040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.467202] Lustre: 68417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1889.549436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1889.554880] Lustre: Skipped 1 previous similar message [ 1889.637945] Lustre: lustre-MDT0001: new disk, initializing [ 1889.740183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.771222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1889.787393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1894.410882] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1904.250593] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1912.133264] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1921.098614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.503038] Lustre: lustre-OST0000: new disk, initializing [ 1921.513454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1921.524878] Lustre: 70389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1921.740803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.788936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1927.800610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1927.940573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1928.658956] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1940.076310] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1945.834175] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1946.163747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1951.749773] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1951.962703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1958.884316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.890789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.894950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.907598] Lustre: Skipped 1 previous similar message [ 1960.871395] Lustre: server umount lustre-OST0000 complete [ 1966.565282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1966.577423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.590810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.599267] Lustre: Skipped 1 previous similar message [ 1969.121431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.124876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.247720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.259501] Lustre: Skipped 2 previous similar messages [ 1978.338414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.355844] Lustre: Skipped 1 previous similar message [ 1979.359323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1979.781044] Lustre: server umount lustre-MDT0000 complete [ 1984.136371] LustreError: 68408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801407 with bad export cookie 9053188813289341817 [ 1984.138117] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.156156] LustreError: 68408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1984.687649] Lustre: server umount lustre-MDT0001 complete [ 1995.460472] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:43:35 (1786801415) [ 2009.698983] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2055.463643] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2067.282129] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2106.615669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.825270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2106.855532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2106.996190] Lustre: lustre-MDT0000: new disk, initializing [ 2107.164591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.192109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2111.545750] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2122.776057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.891307] Lustre: 76427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2122.966909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2122.977273] Lustre: Skipped 1 previous similar message [ 2123.074364] Lustre: lustre-MDT0001: new disk, initializing [ 2123.157026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.205647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2123.218179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2127.308339] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2137.633413] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.134549] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2151.926598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2152.181810] Lustre: lustre-OST0000: new disk, initializing [ 2152.186274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2152.196908] Lustre: 78396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2152.302463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.173945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2159.194317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2159.349762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2159.600176] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2169.806366] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2175.220992] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2175.551467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2180.757209] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2180.903132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2185.192953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.193691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2185.200162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.211602] Lustre: Skipped 1 previous similar message [ 2185.250224] LustreError: Skipped 1 previous similar message [ 2189.219161] Lustre: server umount lustre-OST0000 complete [ 2194.912739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.921200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.926239] Lustre: server umount lustre-MDT0000 complete [ 2200.552177] LustreError: 76439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.564171] LustreError: 76439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2204.002273] LustreError: 76422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801626 with bad export cookie 9053188813289342699 [ 2204.021071] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.021584] LustreError: 76422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2204.375787] Lustre: server umount lustre-MDT0001 complete [ 2213.882082] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:47:14 (1786801634) [ 2221.425218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.739209] LustreError: 80809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.796765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.054079] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2227.177091] LustreError: 80810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.268232] LustreError: 80809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.389051] LustreError: 80810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.678495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.361448] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2250.349149] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2257.147053] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2265.782305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2266.090903] LustreError: 82631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.460686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.467424] Lustre: Skipped 1 previous similar message [ 2272.544731] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2280.170948] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2284.059863] Lustre: 83481:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 05ad6715-6a2a-42b9-9ece-c9b361b371b2 at adminstrative request [ 2287.081287] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.091326] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2297.329432] Lustre: MGS: Client 05ad6715-6a2a-42b9-9ece-c9b361b371b2 (at 0@lo) reconnecting [ 2297.347484] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x7da3634284a6ca5d to 0x7da3634284a6ccfd [ 2297.381697] Lustre: MGC192.168.204.111@tcp: Connection restored to 0@lo (at 0@lo) [ 2321.708547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2321.922148] Lustre: lustre-OST0001: new disk, initializing [ 2321.925972] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2321.932221] Lustre: 83876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2327.893096] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2328.752994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2328.772198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2328.861530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2338.887127] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2344.427090] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2344.431888] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.437211] LustreError: Skipped 1 previous similar message [ 2344.438876] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2344.438882] Lustre: Skipped 4 previous similar messages [ 2344.466324] Lustre: Skipped 2 previous similar messages [ 2356.191439] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2356.469666] Lustre: server umount lustre-OST0001 complete [ 2359.800837] LustreError: 82632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.824583] LustreError: 82632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2363.873546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2363.873985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.885412] LustreError: Skipped 1 previous similar message [ 2363.924248] Lustre: Skipped 1 previous similar message [ 2366.928831] Lustre: server umount lustre-OST0000 complete [ 2380.258231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.265461] Lustre: Skipped 12 previous similar messages [ 2385.378504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2385.616354] Lustre: server umount lustre-MDT0000 complete [ 2389.142627] LustreError: 80789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786801812 with bad export cookie 9053188813289344253 [ 2389.157549] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.160477] LustreError: 80789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2389.533438] Lustre: server umount lustre-MDT0001 complete [ 2398.500304] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:50:18 (1786801818) [ 2400.101608] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2402.257717] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:50:22 (1786801822) [ 2404.027567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2405.996726] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:50:26 (1786801826) [ 2408.003323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2410.035446] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:50:30 (1786801830) [ 2411.838232] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2414.006943] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:50:34 (1786801834) [ 2415.622346] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2421.853183] loop: module loaded [ 2421.887139] loop0: detected capacity change from 0 to 4294967296 [ 2421.928841] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2423.381968] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2426.547460] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2430.995695] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2457.453678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.494118] LustreError: 88004:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2457.503298] LustreError: 88004:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2457.508307] LustreError: 88004:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2457.520088] LustreError: 88004:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2475.495276] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2486.878709] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2522.019945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.163964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2522.181884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2522.234296] Lustre: lustre-MDT0000: new disk, initializing [ 2522.275805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.279699] Lustre: Skipped 1 previous similar message [ 2522.290237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2526.497651] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2537.822381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.962619] Lustre: 90868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2538.227401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2543.077925] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2553.367792] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2559.006327] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2565.022589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.283284] Lustre: 92837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2571.240483] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2571.380834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2571.535794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2580.341701] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2584.907760] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2585.065688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2589.408333] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2589.617118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2592.228095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.236548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.241260] LustreError: Skipped 2 previous similar messages [ 2592.257395] Lustre: Skipped 2 previous similar messages [ 2592.267302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.286477] Lustre: Skipped 3 previous similar messages [ 2597.970269] Lustre: server umount lustre-OST0000 complete [ 2608.579139] Lustre: server umount lustre-MDT0000 complete [ 2612.713560] LustreError: 90874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.728940] LustreError: 90874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2613.167604] LustreError: 90861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802036 with bad export cookie 9053188813289344869 [ 2613.175888] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2613.699510] Lustre: server umount lustre-MDT0001 complete [ 2624.792718] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:54:04 (1786802044) [ 2632.735048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.355814] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2650.943569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.398957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2651.405700] Lustre: Skipped 3 previous similar messages [ 2656.078209] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2664.047128] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2670.899289] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2679.031389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.272132] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2693.795265] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2699.247528] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2705.022783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2710.522345] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2710.886416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2718.553369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.717610] Lustre: lustre-OST0001: new disk, initializing [ 2718.720790] Lustre: Skipped 2 previous similar messages [ 2718.726665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2718.730624] Lustre: Skipped 3 previous similar messages [ 2718.735538] Lustre: 98658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2725.199321] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2727.013893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2727.022774] Lustre: Skipped 2 previous similar messages [ 2727.034045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2727.116732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2735.273543] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2741.135400] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2741.572206] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2748.632457] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2748.883986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2771.009957] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2795.713802] Lustre: Failing over lustre-MDT0000 [ 2796.329374] Lustre: server umount lustre-MDT0000 complete [ 2797.464762] LustreError: 100520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.504594] LustreError: 100520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2799.075210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.089057] Lustre: Skipped 4 previous similar messages [ 2815.456332] Lustre: 66640:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786802222/real 1786802222] req@00000000538a430e x1873596907807744/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786802238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2815.475936] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.655814] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2821.658554] LDISKFS-fs (dm-0): recovery complete [ 2821.663726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.038454] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2829.156520] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2829.156792] Lustre: 95245:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2830.800633] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2831.335031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2831.342478] Lustre: 95248:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2831.354210] Lustre: 95248:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 2831.420540] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2831.753268] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2832.108620] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2832.128043] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2832.456271] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2832.464954] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2832.471848] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2832.486272] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 7 previous similar messages [ 2833.136117] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2833.144330] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 2833.463133] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2833.477437] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 2834.758197] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2834.769395] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 6 previous similar messages [ 2835.414571] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2835.423252] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 2835.752629] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2835.769743] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 2838.954537] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2838.966064] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 12 previous similar messages [ 2839.607945] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2839.619516] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 2839.943277] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2839.954381] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 2846.961616] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2846.988223] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 24 previous similar messages [ 2847.645982] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2847.657584] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 24 previous similar messages [ 2847.975162] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2847.983724] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 24 previous similar messages [ 2863.027988] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2863.046062] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 49 previous similar messages [ 2863.681849] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2863.695898] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 49 previous similar messages [ 2864.023301] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2864.040244] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 49 previous similar messages [ 2889.279304] Lustre: 101508:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2889.295955] Lustre: 101508:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2889.322969] Lustre: 101508:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a1311f32-8b08-41cd-a9ee-84bce62dceea@192.168.204.11@tcp [ 2889.355241] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2889.366440] Lustre: 101508:0:(ldlm_lib.c:1913:abort_req_replay_queue()) @@@ aborted: req@00000000071e9334 x1873596891862400/t0(8589934777) o101->a1311f32-8b08-41cd-a9ee-84bce62dceea@192.168.204.11@tcp:447/0 lens 592/0 e 0 to 0 dl 1786802322 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2889.429821] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2889.462785] Lustre: lustre-MDT0000: Denying connection for new client a1311f32-8b08-41cd-a9ee-84bce62dceea (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2889.476495] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2889.581919] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2889.588566] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2889.605371] Lustre: Skipped 2 previous similar messages [ 2889.680328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2889.683817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2905.568534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.585157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.595090] Lustre: Skipped 6 previous similar messages [ 2910.652384] Lustre: server umount lustre-OST0000 complete [ 2921.364043] Lustre: server umount lustre-OST0001 complete [ 2928.610254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.620309] Lustre: Skipped 6 previous similar messages [ 2940.383148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2941.225490] Lustre: server umount lustre-MDT0000 complete [ 2943.969217] LustreError: 95248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.013451] LustreError: 95248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 2946.025650] LustreError: 98657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802368 with bad export cookie 9053188813289396088 [ 2946.035629] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.055261] LustreError: 98657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2946.608786] Lustre: server umount lustre-MDT0001 complete [ 2955.243466] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:59:35 (1786802375) [ 2963.743616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.749166] Lustre: *** cfs_fail_loc=197, val=0*** [ 2979.359652] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2984.481265] LustreError: 103483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 2989.630597] Lustre: *** cfs_fail_loc=197, val=0*** [ 2989.634650] Lustre: Skipped 65 previous similar messages [ 3010.208634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.222263] Lustre: Skipped 3 previous similar messages [ 3016.358173] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3023.411832] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3025.519902] Lustre: server umount lustre-OST0000 complete [ 3033.596863] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:00:54 (1786802454) [ 3055.326420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.058468] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3070.772247] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3074.774488] LustreError: 105146:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3074.789470] LustreError: 105146:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3074.794283] LustreError: 105146:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3075.047819] Lustre: server umount lustre-MDT0000 complete [ 3107.702929] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3117.586697] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3154.495677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.723852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3154.752726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3154.847420] Lustre: lustre-MDT0000: new disk, initializing [ 3154.917554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3160.298110] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3170.000232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.116949] Lustre: 109321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3170.319958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3174.511506] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3184.006900] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3189.085119] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3195.956945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.284952] Lustre: 111292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3201.823648] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3204.263613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3210.934532] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3216.520457] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3216.859613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3221.709515] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3221.963553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3225.060134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.070073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.072930] Lustre: Skipped 1 previous similar message [ 3225.079415] Lustre: Skipped 13 previous similar messages [ 3230.569489] Lustre: server umount lustre-OST0000 complete [ 3241.192293] Lustre: server umount lustre-MDT0000 complete [ 3245.541709] LustreError: 109334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.568550] LustreError: 109334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3245.944994] LustreError: 109315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802668 with bad export cookie 9053188813289406742 [ 3245.950427] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3245.952042] LustreError: 109315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3246.587078] Lustre: server umount lustre-MDT0001 complete [ 3258.066752] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:04:37 (1786802677) [ 3268.461460] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3271.482917] Key type lgssc unregistered [ 3271.735403] LNet: 113778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3271.740515] LNetError: 113778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3271.760350] LNet: Removed LNI 192.168.204.111@tcp [ 3272.595230] Key type .llcrypt unregistered [ 3272.597259] Key type ._llcrypt unregistered [ 3291.050302] Key type ._llcrypt registered [ 3291.052798] Key type .llcrypt registered [ 3291.137952] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3307.752157] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3308.661274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3308.787346] alg: No test for adler32 (adler32-zlib) [ 3309.945550] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 3310.163506] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3311.895641] Key type lgssc registered [ 3313.219701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3357.300484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3357.329845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.563443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3358.594308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3358.665712] Lustre: lustre-MDT0000: new disk, initializing [ 3358.744436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.761744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3362.412823] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3373.145417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3373.597069] Lustre: lustre-OST0000: new disk, initializing [ 3373.606418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3373.612472] Lustre: Skipped 1 previous similar message [ 3373.622476] Lustre: 117953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3373.766370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.885242] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3381.842681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3381.854446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3381.886594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3392.705105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3392.859145] Lustre: lustre-OST0001: new disk, initializing [ 3392.865261] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3392.878909] Lustre: 118816:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3392.972636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.529952] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3401.819955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3401.830394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3401.873580] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3411.941631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.949381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.965065] Lustre: Skipped 1 previous similar message [ 3411.986295] Lustre: Skipped 1 previous similar message [ 3415.948092] Lustre: server umount lustre-MDT0000 complete [ 3424.913646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3430.819149] LustreError: 117137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786802853 with bad export cookie 1264219075585354268 [ 3430.847786] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.850570] LustreError: 117137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3431.012510] Lustre: server umount lustre-OST0000 complete [ 3435.676444] Lustre: server umount lustre-OST0001 complete [ 3446.483571] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:07:46 (1786802866) [ 3472.954573] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:08:12 (1786802892) [ 3483.977787] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3510.549950] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:08:50 (1786802930) [ 3551.534055] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3562.999294] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3602.259362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.490912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3602.523332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3602.624829] Lustre: lustre-MDT0000: new disk, initializing [ 3602.757830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.784898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3607.437908] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3618.874828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.128976] Lustre: 126575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3619.317502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3619.322986] Lustre: Skipped 1 previous similar message [ 3619.495384] Lustre: lustre-MDT0001: new disk, initializing [ 3619.703662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3619.772563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3619.790689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3624.495794] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3636.143622] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3643.477904] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3651.306770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.620974] Lustre: lustre-OST0000: new disk, initializing [ 3651.625413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3651.633919] Lustre: 128541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3651.724754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3657.814487] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3660.337938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3660.352678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3660.436838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3668.146385] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3673.887992] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3674.272864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3678.673720] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3678.836569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3681.249402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3681.249424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.261368] LustreError: Skipped 1 previous similar message [ 3681.262480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.269195] Lustre: Skipped 1 previous similar message [ 3681.284345] Lustre: Skipped 1 previous similar message [ 3686.375772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3686.383983] Lustre: Skipped 1 previous similar message [ 3686.833783] Lustre: server umount lustre-OST0000 complete [ 3691.488937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.494127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3691.516297] Lustre: Skipped 1 previous similar message [ 3691.523867] Lustre: Skipped 1 previous similar message [ 3696.608168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.612780] Lustre: Skipped 1 previous similar message [ 3696.934853] Lustre: server umount lustre-MDT0000 complete [ 3700.248668] LustreError: 126566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803123 with bad export cookie 1264219075585355493 [ 3700.257427] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3700.265083] LustreError: 126566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3700.769027] Lustre: server umount lustre-MDT0001 complete [ 3708.448905] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:12:09 (1786803129) [ 3715.447164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.147686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.507133] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3721.188393] LustreError: 130952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.216485] LustreError: 130952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3726.311130] LustreError: 130953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.146024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.424854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3736.573736] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3744.627233] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3751.200243] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3758.368543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3758.734351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.742770] Lustre: Skipped 1 previous similar message [ 3765.961448] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3773.467475] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3781.202151] Lustre: *** cfs_fail_loc=159, val=0*** [ 3781.219485] Lustre: Skipped 1 previous similar message [ 3797.403336] Lustre: lustre-MDT0000: Client e4b633e2-4634-4818-aa19-827cf53dc454 (at 192.168.204.11@tcp) reconnecting [ 3799.208151] Lustre: *** cfs_fail_loc=159, val=0*** [ 3799.211549] Lustre: Skipped 2 previous similar messages [ 3815.848564] Lustre: lustre-MDT0000: Client e4b633e2-4634-4818-aa19-827cf53dc454 (at 192.168.204.11@tcp) reconnecting [ 3822.575751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.580274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.589039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.605864] Lustre: Skipped 1 previous similar message [ 3822.859809] Lustre: server umount lustre-OST0000 complete [ 3827.682304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.706169] Lustre: Skipped 1 previous similar message [ 3833.085849] Lustre: server umount lustre-MDT0000 complete [ 3836.700061] LustreError: 130933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803259 with bad export cookie 1264219075585356368 [ 3836.701692] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.704847] LustreError: 130933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3836.967411] Lustre: server umount lustre-MDT0001 complete [ 3849.179613] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3852.476201] Key type lgssc unregistered [ 3852.819850] LNet: 134736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3852.824872] LNetError: 134736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3852.841277] LNet: Removed LNI 192.168.204.111@tcp [ 3853.593768] Key type .llcrypt unregistered [ 3853.595922] Key type ._llcrypt unregistered [ 3876.874592] Key type ._llcrypt registered [ 3876.877061] Key type .llcrypt registered [ 3876.981345] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3877.823873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3877.890199] alg: No test for adler32 (adler32-zlib) [ 3879.058463] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 3879.419117] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3881.087448] Key type lgssc registered [ 3882.270039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3889.679291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3889.698829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.100400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.330425] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3901.411184] LustreError: 136152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.431494] LustreError: 136152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3905.949125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.492242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.923526] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3922.370951] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3928.928667] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3935.598356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.200087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.211966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3942.275576] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3949.197266] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3961.909836] Lustre: *** cfs_fail_loc=159, val=0*** [ 3978.121856] Lustre: lustre-MDT0000: Client 0e2cfa7f-1f46-4c1d-881c-ed2768b57df9 (at 192.168.204.11@tcp) reconnecting [ 3980.355798] Lustre: *** cfs_fail_loc=159, val=0*** [ 3996.627695] Lustre: lustre-MDT0001: Client 0e2cfa7f-1f46-4c1d-881c-ed2768b57df9 (at 192.168.204.11@tcp) reconnecting [ 3998.107146] Lustre: *** cfs_fail_loc=159, val=0*** [ 3998.118806] Lustre: Skipped 3 previous similar messages [ 4014.480829] Lustre: lustre-MDT0001: Client 0e2cfa7f-1f46-4c1d-881c-ed2768b57df9 (at 192.168.204.11@tcp) reconnecting [ 4018.788474] Lustre: 139191:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4021.604581] Lustre: *** cfs_fail_loc=159, val=0*** [ 4021.606218] Lustre: Skipped 4 previous similar messages [ 4038.082720] Lustre: lustre-MDT0000: Client c304f914-ea5f-4d37-a68d-05d185bc9ac6 (at 192.168.204.11@tcp) reconnecting [ 4039.892937] Lustre: *** cfs_fail_loc=159, val=0*** [ 4039.899985] Lustre: Skipped 14 previous similar messages [ 4055.471768] Lustre: lustre-MDT0000: Client c304f914-ea5f-4d37-a68d-05d185bc9ac6 (at 192.168.204.11@tcp) reconnecting [ 4057.358680] Lustre: 139486:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4079.584497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.592210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.600141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.632731] Lustre: Skipped 1 previous similar message [ 4082.252103] Lustre: server umount lustre-OST0000 complete [ 4089.826281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.836116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.836842] Lustre: Skipped 1 previous similar message [ 4089.845967] Lustre: Skipped 2 previous similar messages [ 4092.240363] Lustre: server umount lustre-MDT0000 complete [ 4094.946198] LustreError: 136152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.957322] LustreError: 136152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4095.029930] LustreError: 136133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803517 with bad export cookie 3088709832286572942 [ 4095.032799] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.046807] LustreError: 136133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4095.274750] Lustre: server umount lustre-MDT0001 complete [ 4103.438616] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4105.680619] Key type lgssc unregistered [ 4105.966791] LNet: 140562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4105.981095] LNetError: 140562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4106.000945] LNet: Removed LNI 192.168.204.111@tcp [ 4106.665810] Key type .llcrypt unregistered [ 4106.673094] Key type ._llcrypt unregistered [ 4131.640746] Key type ._llcrypt registered [ 4131.642743] Key type .llcrypt registered [ 4131.715507] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4132.695237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4132.787890] alg: No test for adler32 (adler32-zlib) [ 4133.883743] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4134.212680] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4135.927669] Key type lgssc registered [ 4137.983133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.368427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.422678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.115448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.968822] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4159.460477] LustreError: 141978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.489739] LustreError: 141978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4162.762577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.039323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.952723] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4175.720331] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4181.126447] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4188.376964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4188.720584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.153674] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4194.871341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4202.042712] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4207.154228] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4212.131562] Lustre: 144560:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4215.575919] Lustre: 144608:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4217.185473] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4218.248717] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4221.309540] Lustre: 144755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4222.210690] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4223.207652] Lustre: 144853:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4227.559157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.570783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4227.571330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4227.579554] Lustre: Skipped 1 previous similar message [ 4232.677262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.684570] Lustre: Skipped 2 previous similar messages [ 4237.798449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4237.804471] Lustre: Skipped 1 previous similar message [ 4241.375868] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4241.535713] Lustre: server umount lustre-OST0000 complete [ 4248.035080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.052531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.062450] Lustre: Skipped 1 previous similar message [ 4248.077429] Lustre: Skipped 1 previous similar message [ 4251.798717] Lustre: server umount lustre-MDT0000 complete [ 4253.154588] LustreError: 141978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.222438] LustreError: 144657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803679 with bad export cookie 4193823782361132509 [ 4256.230343] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4256.238922] LustreError: 144657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4256.761644] Lustre: server umount lustre-MDT0001 complete [ 4269.340556] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4271.886366] Key type lgssc unregistered [ 4272.146244] LNet: 145915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4272.155111] LNetError: 145915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4273.191939] LNet: Removed LNI 192.168.204.111@tcp [ 4274.112227] Key type .llcrypt unregistered [ 4274.115769] Key type ._llcrypt unregistered [ 4298.086891] Key type ._llcrypt registered [ 4298.089819] Key type .llcrypt registered [ 4298.186842] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4299.338343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4299.379117] alg: No test for adler32 (adler32-zlib) [ 4300.436355] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4300.714573] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4302.407203] Key type lgssc registered [ 4303.482732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4310.968535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4311.006573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.519392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.076364] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4322.786704] LustreError: 147331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.803608] LustreError: 147331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4326.022749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.757098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.752389] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4339.782713] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4345.608278] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.973541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4352.434490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.574336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4358.467220] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4367.546476] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4371.973770] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4377.527895] Lustre: *** cfs_fail_loc=159, val=0*** [ 4393.904075] Lustre: lustre-MDT0000: Client 545017b6-9c3c-4897-9ad1-9faf2f24c6fe (at 192.168.204.11@tcp) reconnecting [ 4412.899371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4412.906949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.910985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4412.931953] Lustre: Skipped 1 previous similar message [ 4416.545694] Lustre: server umount lustre-OST0000 complete [ 4423.137326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.146169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.156740] Lustre: Skipped 1 previous similar message [ 4423.169354] Lustre: Skipped 2 previous similar messages [ 4427.275985] Lustre: server umount lustre-MDT0000 complete [ 4428.275529] LustreError: 147331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.305553] LustreError: 147331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4431.128051] LustreError: 147310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786803854 with bad export cookie 10057728902383923859 [ 4431.133931] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.144489] LustreError: 147310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4431.686421] Lustre: server umount lustre-MDT0001 complete [ 4442.816333] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4445.274726] Key type lgssc unregistered [ 4445.529792] LNet: 150997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.539424] LNetError: 150997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4445.561423] LNet: Removed LNI 192.168.204.111@tcp [ 4446.401809] Key type .llcrypt unregistered [ 4446.404904] Key type ._llcrypt unregistered [ 4469.867141] Key type ._llcrypt registered [ 4469.869214] Key type .llcrypt registered [ 4469.942212] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4470.639501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4470.779080] alg: No test for adler32 (adler32-zlib) [ 4471.841414] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4472.140079] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4473.890423] Key type lgssc registered [ 4475.112622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4482.989123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4483.024074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.435324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.433863] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4494.823285] LustreError: 152414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.842442] LustreError: 152414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4496.679116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.954375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4500.946334] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4508.440788] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4514.132357] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4521.625391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4522.166940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.087204] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4529.331867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4535.177315] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4541.657787] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4558.909388] Lustre: lustre-MDT0001: evicting 008f1070-7d8e-4e15-bd66-055345fa5300 (at 192.168.204.11@tcp) by administrative request [ 4558.919315] Lustre: Skipped 1 previous similar message [ 4588.000642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.005069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.015227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.045794] Lustre: Skipped 1 previous similar message [ 4593.125170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.134205] Lustre: Skipped 1 previous similar message [ 4598.246120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4598.251681] Lustre: Skipped 1 previous similar message [ 4602.335294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4602.529663] Lustre: server umount lustre-OST0000 complete [ 4608.480774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.482663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.510913] Lustre: Skipped 1 previous similar message [ 4608.523825] Lustre: Skipped 1 previous similar message [ 4613.016369] Lustre: server umount lustre-MDT0000 complete [ 4613.605942] LustreError: 153467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.630348] LustreError: 153467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4618.266141] LustreError: 152393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804041 with bad export cookie 4043429428330141908 [ 4618.268325] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4618.284044] LustreError: 152393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4618.803512] Lustre: server umount lustre-MDT0001 complete [ 4630.079633] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4633.034082] Key type lgssc unregistered [ 4633.301670] LNet: 156482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4633.306631] LNetError: 156482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4633.333809] LNet: Removed LNI 192.168.204.111@tcp [ 4634.042258] Key type .llcrypt unregistered [ 4634.046580] Key type ._llcrypt unregistered [ 4710.563896] Key type ._llcrypt registered [ 4710.566827] Key type .llcrypt registered [ 4710.691995] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4712.147198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4712.187592] alg: No test for adler32 (adler32-zlib) [ 4713.296572] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4713.632729] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4715.367245] Key type lgssc registered [ 4716.366503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4723.845191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4723.878072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.448789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.394438] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4735.971378] LustreError: 158973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.003679] LustreError: 158973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4740.376693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.782430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4745.825824] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4756.811372] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4762.556699] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4769.672827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4770.024537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.056866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4775.878276] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4783.365959] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4785.411550] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4791.777082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4791.782168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.798672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.875147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4795.875888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.883492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.896216] Lustre: Skipped 1 previous similar message [ 4797.763303] Lustre: server umount lustre-OST0000 complete [ 4801.007376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.010758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4801.016280] Lustre: Skipped 1 previous similar message [ 4801.026659] Lustre: Skipped 1 previous similar message [ 4806.117418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4806.122942] Lustre: Skipped 1 previous similar message [ 4811.233802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.253943] Lustre: Skipped 1 previous similar message [ 4815.328076] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4815.643182] Lustre: server umount lustre-MDT0000 complete [ 4819.697027] LustreError: 158953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804242 with bad export cookie 12454463788244295375 [ 4819.712089] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.715292] LustreError: 158953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4820.091974] Lustre: server umount lustre-MDT0001 complete [ 4833.142903] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4835.691629] Key type lgssc unregistered [ 4835.922953] LNet: 162518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4835.927932] LNetError: 162518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4835.938056] LNet: Removed LNI 192.168.204.111@tcp [ 4837.122515] Key type .llcrypt unregistered [ 4837.128015] Key type ._llcrypt unregistered [ 4864.929586] Key type ._llcrypt registered [ 4864.931988] Key type .llcrypt registered [ 4865.096573] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4866.303765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4866.321280] alg: No test for adler32 (adler32-zlib) [ 4867.295567] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 4867.481899] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4869.143359] Key type lgssc registered [ 4870.382253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4886.061940] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:31:46 (1786804306) [ 4902.607478] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4915.072426] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4956.587711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4956.647206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.924272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4957.965899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4958.032375] Lustre: lustre-MDT0000: new disk, initializing [ 4958.195633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4958.245805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4963.366060] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4974.189280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.364241] Lustre: 166782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4974.443987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4974.456163] Lustre: Skipped 1 previous similar message [ 4974.566196] Lustre: lustre-MDT0001: new disk, initializing [ 4974.699408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4974.757913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4974.783019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4979.139569] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4988.445762] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4993.777321] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4999.625780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4999.953926] Lustre: lustre-OST0000: new disk, initializing [ 4999.962946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4999.970679] Lustre: 168750:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5000.078188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.544540] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5007.920667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5007.944149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5008.043182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5014.191976] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5028.851456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5028.864819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.882430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5028.889697] Lustre: Skipped 1 previous similar message [ 5030.946598] Lustre: server umount lustre-OST0000 complete [ 5036.004517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5036.015892] LustreError: Skipped 1 previous similar message [ 5036.020957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.036890] Lustre: Skipped 1 previous similar message [ 5036.048774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5039.074380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.077265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5039.100802] Lustre: Skipped 1 previous similar message [ 5044.193893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.204789] Lustre: Skipped 1 previous similar message [ 5048.289543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5048.296407] Lustre: Skipped 1 previous similar message [ 5049.311236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5049.473420] Lustre: server umount lustre-MDT0000 complete [ 5054.302679] LustreError: 166776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786804477 with bad export cookie 12702573658134234729 [ 5054.312132] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.330449] LustreError: 166776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5054.595354] Lustre: server umount lustre-MDT0001 complete [ 5067.224119] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5070.235538] Key type lgssc unregistered [ 5070.564419] LNet: 170658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5070.572149] LNetError: 170658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5070.590646] LNet: Removed LNI 192.168.204.111@tcp [ 5071.629743] Key type .llcrypt unregistered [ 5071.631914] Key type ._llcrypt unregistered [ 5100.739675] Key type ._llcrypt registered [ 5100.743702] Key type .llcrypt registered [ 5100.883114] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5101.758812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5101.792868] alg: No test for adler32 (adler32-zlib) [ 5102.934912] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 5103.138825] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5104.801131] Key type lgssc registered [ 5106.158187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5113.299165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5113.314766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.759850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.248313] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5124.068188] LustreError: 172074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.087453] LustreError: 172074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5129.124555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.194743] LustreError: 172073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.579861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.245757] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5144.747339] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5152.438548] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5159.480503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5159.779142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.215615] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5176.581695] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5184.372277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5193.086432] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5193.105462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.114819] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5193.127166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.141459] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5194.030358] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5194.052473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.065449] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5194.073686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.084335] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5194.771262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.667724] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5195.682129] Lustre: Skipped 1 previous similar message [ 5195.701417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.723380] Lustre: Skipped 1 previous similar message [ 5195.734592] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5195.747212] Lustre: Skipped 1 previous similar message [ 5195.753073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.772738] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5195.783097] Lustre: Skipped 1 previous similar message [ 5196.939634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.886359] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5197.894593] Lustre: Skipped 1 previous similar message [ 5197.912980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.926367] Lustre: Skipped 1 previous similar message [ 5197.933438] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5197.941305] Lustre: Skipped 1 previous similar message [ 5197.952467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.967504] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5197.979580] Lustre: Skipped 1 previous similar message [ 5199.061667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.942349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.909868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.800089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.866876] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5202.882655] Lustre: Skipped 4 previous similar messages [ 5202.907829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.916455] Lustre: Skipped 4 previous similar messages [ 5202.920234] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5202.924715] Lustre: Skipped 4 previous similar messages [ 5202.928155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.945511] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5202.957463] Lustre: Skipped 4 previous similar messages [ 5203.804613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.768064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.711234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.825418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.659908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.505506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.428228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.515489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.271625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5211.276810] Lustre: Skipped 8 previous similar messages [ 5211.292613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.304193] Lustre: Skipped 8 previous similar messages [ 5211.309828] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5211.313520] Lustre: Skipped 8 previous similar messages [ 5211.316731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.325670] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5211.333167] Lustre: Skipped 8 previous similar messages [ 5212.211065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.135405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.133881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.861583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.813892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.703333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.791972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.719087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.623171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.453098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.540621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.519430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.408340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.346720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.067396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.986733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.008815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.232842] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5228.238177] Lustre: Skipped 17 previous similar messages [ 5228.266065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.283766] Lustre: Skipped 17 previous similar messages [ 5228.289052] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5228.294687] Lustre: Skipped 17 previous similar messages [ 5228.299919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.309239] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5228.315620] Lustre: Skipped 17 previous similar messages [ 5229.252870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.144824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.003951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.915304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.752899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.706719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.501153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.385108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.236205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.306823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.112575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.942173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.779911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.612734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.505746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.444651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.377161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.365919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.754518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.001623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.865530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.795476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.812928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.076573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.810495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.816924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.851215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.794365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.917339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.162804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.314822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.421160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.603841] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5260.609441] Lustre: Skipped 32 previous similar messages [ 5260.626530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.639623] Lustre: Skipped 32 previous similar messages [ 5260.647347] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5260.655304] Lustre: Skipped 32 previous similar messages [ 5260.661080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.681067] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5260.689266] Lustre: Skipped 32 previous similar messages [ 5261.804069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.836124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.901160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.005299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.973926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.211995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.187561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.120603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.189793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.209247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.097752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.317568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.365974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.402475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.505749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.475486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.433920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.147512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.110938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.218113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.197864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.253145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.435261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.340737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.515412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.626866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.756602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.980611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.055136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.960514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.957396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.728470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.621441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.340487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.301556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.232054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.118648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.113347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.122513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.944840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.827134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.728250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.623137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.521133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.469536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.248898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.082983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.033743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.997700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.062989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.867469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.679731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.817235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.699680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.616914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.554501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.435541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.115718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.203322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.444275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.509953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.479533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.442883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.417320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.646895] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5324.651095] Lustre: Skipped 64 previous similar messages [ 5324.665663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.673695] Lustre: Skipped 64 previous similar messages [ 5324.678043] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5324.684626] Lustre: Skipped 64 previous similar messages [ 5324.687804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.698189] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5324.705793] Lustre: Skipped 64 previous similar messages [ 5325.632769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.767773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.668802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.702602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.571766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.498833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.422361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.269334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.129655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.988899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.061663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.016789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.035869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.984696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.955166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.919917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.920045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.787264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.920126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.869355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.114114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.211198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.153260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.111436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.046078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.068136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.163092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.081840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.093946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.995209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.936943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.699632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.589669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.546638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.595870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.668168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.491890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.505108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.525668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.380827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.371808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.329987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.398384] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5366.417457] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5366.423777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.182562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.235412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.154745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.150940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.162805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.246258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.268780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.297132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.145739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.330975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.427069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.501479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.233363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.290350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.130916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.010656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.036739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.074519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.977478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.794593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.676408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.701968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.678677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.628604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.333569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.280673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.126636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.178978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.269831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.383710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.577515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.591675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.564890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.649974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.495791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.430563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.331972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.143931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.032202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.922652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.785920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.652299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.406716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.483151] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5408.493706] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5408.518551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.642375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.560463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.573553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.534312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.477623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.506797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.541963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.525407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.608896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.631945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.675452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.492643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.547928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.694383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.432381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.556562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.795568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.854801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.850309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.992907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.104392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.049675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.857916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.941560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.795698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.609316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.332525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.199171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.081367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.961339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.791055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.683573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.543146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.363175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.196260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.015745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.839818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.954301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.782510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.811657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.762219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.032574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.977809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.846792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.749586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.620313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.384804] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5453.388811] Lustre: Skipped 133 previous similar messages [ 5453.401391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.412550] Lustre: Skipped 133 previous similar messages [ 5453.415905] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5453.422184] Lustre: Skipped 133 previous similar messages [ 5453.425886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.434391] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5453.440389] Lustre: Skipped 133 previous similar messages [ 5454.212178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.905331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.947882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.924329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.908502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.764969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.691793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.524307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.339815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.239493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.316184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.187149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.176145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.155612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.296822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.253141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.270443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.244924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.335655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.126443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.120296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.927499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.929390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.783630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.683739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.533589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.482561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.444826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.469764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.158529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.964093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.780649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.708793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.584881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.574772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.595719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.628284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.589526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.655068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.416968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.433690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.454644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.571695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.687608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.683685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.720653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.671708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.682830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.595034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.600444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.520683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.316115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.409069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.475971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.579762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.472114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.390488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.360469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.380955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.294911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.367481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.499364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.552219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.453354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.643336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.565188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.615863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.663885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.529857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.535695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.561429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.815777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.847586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.892850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.880956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.115153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.129829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.125290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.043808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.815231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.776837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.741666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.646545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.677296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.550024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.798832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.740927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.750488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.449096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.587331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.950200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.013981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.055237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.487800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.686423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.675813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.740362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.789863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.907313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.031439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.385853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.673978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.850317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.839122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.774440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.907044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.991186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.136950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.108038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.320853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.345624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.195457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.406137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.244564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.128126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.028493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.898756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.729856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.775509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.779609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.900156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.743487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.796718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.630654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.879280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.832669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.906861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.060533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.047967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.293121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.279790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.419960] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5585.435883] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5585.458702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.555980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.411941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.389536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.553803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.775795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.748592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.732924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.639421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.805728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.869969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.868796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.864557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.026656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.221842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.474797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.576931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.389652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.459626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.347079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.343952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.443151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.589923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.399903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.496935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.419733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.598576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.727938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.805583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.235397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.500011] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.651522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.714542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.022886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.320105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.361236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.392684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.453820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.444966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.708111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.725638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.909310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.666269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.608383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.461683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.737527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.584239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.787790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.792693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.886361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.738987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.724348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.616315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.708747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.870455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.756600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.735896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.570175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.625621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.596459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.425217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.237401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.066253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.212720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.344539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.262787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.135756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.217109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.352995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.333959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.434403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.479516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.414964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.466081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.622706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.553495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.890978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.984376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.127107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.278653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.286247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.359593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.457522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.528779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.434954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.658546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.808068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.845442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.617010] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.649605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.640839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.580835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.536987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.575333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.436784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.406116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.387274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.292712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.143416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.143752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.113169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.125404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.128698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.089314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.901560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.816948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.683366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.634466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.560807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.725731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.821250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.701687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.598952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.527770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.579578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.387345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.454941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.193637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.131862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.995356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.088067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.930458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.112261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5710.127953] Lustre: Skipped 253 previous similar messages [ 5710.156486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.171867] Lustre: Skipped 253 previous similar messages [ 5710.175454] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5710.181487] Lustre: Skipped 253 previous similar messages [ 5710.184925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.195033] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5710.201688] Lustre: Skipped 253 previous similar messages [ 5711.163769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.357220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.281413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.377770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.362547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.336499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.352513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.363325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.427459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.449796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.644964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.713183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.866843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.981419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.088261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.166044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.198666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.390987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.515751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.880801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.819994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.795545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.751758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.910304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.080791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.259800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.465844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.474091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.524600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.401384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.288921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.245309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.282795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.109275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.055326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.946414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.986371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.807914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.752922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.623050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.516768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.345088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.389570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.173775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.408495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.337942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.396423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.254912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.314540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.478253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.787974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.968183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.898912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.008669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.945231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.943807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.880816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.933372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.682064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.869523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.877256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.877319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.864106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.746735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.632478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.742254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.679868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.697709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.776845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.600212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.611505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.516602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.773635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.865760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.886684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.984232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.167757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.261707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.445503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.469537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.702280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.984210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.868441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.215442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.139954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.408083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.596213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.864649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.090586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.902142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.968823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.823670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.369398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.651194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.141403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.608786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.351509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.357073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.276776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.616950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.158923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.497134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.636123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.608198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.369705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.287598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.172036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.041431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.592353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.822214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.040873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.201784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.352217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.609708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.714189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.839228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.083516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.464268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.956995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.822794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.802971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.324510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.798584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.378158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.837780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.691480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.461446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.590752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.979157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.048757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.051197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.059810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.055205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.168731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.088086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.090278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.553885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.371429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.271893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.179132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.413992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.560236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.731192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.852166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.036686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.545502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.877225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.139442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.265109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.440366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.506698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.476369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.588523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.730259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.681544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.806659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.729075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.678257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.639750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.689495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.600568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.911228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.972211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.916920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.850044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.765246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.161947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.006944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.090761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.124863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.040596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.976970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.247412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.267976] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5900.284705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.291804] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5901.176691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.052995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.157557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.180495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.212785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.252137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.156654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.000652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.760600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.691913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.638730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.624453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.665576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.751562] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5913.768298] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5913.775297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.997052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.083703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.269579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.177726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.166296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.212328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.389374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.307356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.464448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.487806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.503241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.290765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.300305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.195864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.350405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.108686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.232241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.185812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.242427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.395205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.565970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.073095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.888387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.039307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.000987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.933894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.861575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.886071] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 5942.899970] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 5942.904651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.032690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.311248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.003662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.858807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.806759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.828903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.603900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.601309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.508072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.622228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.582532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.676987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.632648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.775297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.615824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.658769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.546797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.665666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.654740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.539416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.399759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.366896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.410594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.339261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.271798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.169642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.112376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.071534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.758455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.775511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.617406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.742962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.703961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.580121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.375821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.206829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.028124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.001468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.002533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.091525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.065620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.966292] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5982.977057] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 2 previous similar messages [ 5982.981140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.987362] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5982.987740] LustreError: 172093:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 2 previous similar messages [ 5984.085324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.171992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.956812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.887217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.931251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.692640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.705861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.788113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.833946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.771938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.508232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.654431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.610526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.632404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.617395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.395340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.127628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.191763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.328468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.360173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.193627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.118483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.083768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.170603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.976265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.831276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.719943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.540180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.499239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.560294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.196814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.066626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.840691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.739475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.764593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.835355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.744996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.723442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.707271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.731698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.693798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.625267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.467702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.463290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.251738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.278274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.123930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.083552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.996503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.967602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.739906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.725373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.555852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.678633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.536804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.448200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.290802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.305504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.179934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.097962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.907556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.774871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.594630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.471429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.506542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.469933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.405570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.431894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.523755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.516753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.494768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.270230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.163994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.139920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.202866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.200928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.341673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.193953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.011089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.766851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.436343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.374677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.519081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.393588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.517564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.384630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.748100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.824846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.047212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.030816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.152820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.088180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.114761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.144637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.121546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.940577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.029919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.053330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.823581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.691919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.534091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.549834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.531754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.360980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.283126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.165613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.088914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.893816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.704037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.496586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.415961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.212430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.101225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.871361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.731470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.620496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.574318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.519143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.499387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.475635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.322877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.141172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.992816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.802693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.623731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.524809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.428068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.436160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.261791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.252456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.298085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.942890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.761872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.504656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.520660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.290055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.092132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.808985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.787896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.699094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.663315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.601663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.550293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.631089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.754033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.685080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.511626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.298980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.660788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.535537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.456674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.152072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.042806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.088873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.929664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.764537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.682700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.689431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.606205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.585348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.594911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.708572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.808633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.139780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.150227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.253961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.082912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.303368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.426339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.439047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.289101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.448828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.778052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.876500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.956277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.266662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.778241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.941921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.762483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.996915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.099116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.951676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.817203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.017093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.072722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.236979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.238292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.347908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.463699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.549294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.779086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.905253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.085418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.010953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.266541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.119852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.188059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.179266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.033649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.253719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.568869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.539958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.683150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.099324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.752629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.691338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.877844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.870272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.881693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.079678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.306584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.436442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.511798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.962055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.895313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.011861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.235497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.609354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.171890] LustreError: 173894:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6614.181215] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6614.189623] LustreError: 172093:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 7349.218408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.227466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.255213] Lustre: Skipped 477 previous similar messages [ 7349.287609] Lustre: Skipped 1 previous similar message [ 7354.375539] LustreError: 226174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.418219] LustreError: 226174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7354.601850] Lustre: server umount lustre-OST0000 complete [ 7362.023651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7362.031094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7364.581255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.683758] Lustre: server umount lustre-MDT0000 complete [ 7369.699252] LustreError: 172070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.721386] LustreError: 172070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7370.977760] LustreError: 172053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786806793 with bad export cookie 8651350243297717655 [ 7370.988582] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.994607] LustreError: 172053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7371.363549] Lustre: server umount lustre-MDT0001 complete [ 7383.948054] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7386.460578] Key type lgssc unregistered [ 7386.732637] LNet: 227384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7386.746337] LNetError: 227384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7387.814988] LNet: Removed LNI 192.168.204.111@tcp [ 7388.717243] Key type .llcrypt unregistered [ 7388.725602] Key type ._llcrypt unregistered [ 7419.374591] Key type ._llcrypt registered [ 7419.376977] Key type .llcrypt registered [ 7419.499517] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7420.699808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7420.809370] alg: No test for adler32 (adler32-zlib) [ 7421.938166] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 7422.178552] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7423.935225] Key type lgssc registered [ 7425.093814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7432.507758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7432.521472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.989911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.395819] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7443.427896] LustreError: 228801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.445072] LustreError: 228801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7448.545785] LustreError: 228802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.690286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.315868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7454.506728] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7464.659876] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7471.554731] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7478.648335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7478.990412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.923901] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7493.461462] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7503.629608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7508.448689] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7510.370118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.383992] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7510.394095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7510.403536] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7510.403893] LustreError: 228821:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7514.084645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.100318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.013838] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 7515.018314] Lustre: Skipped 1 previous similar message [ 7518.724542] Lustre: server umount lustre-OST0000 complete [ 7530.455181] LustreError: 228821:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7553.042937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.348820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.356290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7554.373461] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7554.375224] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7554.400816] Lustre: Skipped 1 previous similar message [ 7560.007501] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7567.719549] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7569.369337] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7576.359215] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fb284c4d000.ost_server_uuid 50 [ 7578.290822] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9fb284c4d000.ost_server_uuid in IDLE state after 0 sec [ 7583.714680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7583.737279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.764212] Lustre: Skipped 1 previous similar message [ 7583.775321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.653434] Lustre: server umount lustre-OST0000 complete [ 7595.489470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7595.492886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7595.523596] Lustre: Skipped 2 previous similar messages [ 7595.523739] Lustre: Skipped 2 previous similar messages [ 7599.444869] Lustre: server umount lustre-MDT0000 complete [ 7600.619611] LustreError: 228801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.462548] LustreError: 230646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807026 with bad export cookie 67193557880444405 [ 7603.465120] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.471370] LustreError: 230646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7603.841071] Lustre: server umount lustre-MDT0001 complete [ 7615.363771] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7618.915186] Key type lgssc unregistered [ 7619.379806] LNet: 234039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7619.388178] LNetError: 234039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7619.412870] LNet: Removed LNI 192.168.204.111@tcp [ 7620.476812] Key type .llcrypt unregistered [ 7620.481703] Key type ._llcrypt unregistered [ 7654.303480] Key type ._llcrypt registered [ 7654.308184] Key type .llcrypt registered [ 7654.444229] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7655.600969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7655.701844] alg: No test for adler32 (adler32-zlib) [ 7656.720633] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 7656.979708] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7658.775572] Key type lgssc registered [ 7660.214183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7670.080864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7670.094789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.434194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.634301] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7682.019450] LustreError: 235770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.035651] LustreError: 235770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7684.691533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.128117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7689.195964] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7692.424687] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7699.639481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.056736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.629583] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7716.357036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.547411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7716.551644] Lustre: lustre-OST0001: new disk, initializing [ 7716.569508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7716.578972] Lustre: 238078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7716.700930] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7722.078110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7722.098269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7722.179526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7722.620359] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7732.832614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7740.743953] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7753.192775] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.195271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7753.204385] Lustre: Skipped 2 previous similar messages [ 7753.213951] Lustre: Skipped 1 previous similar message [ 7758.311476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7758.319793] Lustre: Skipped 5 previous similar messages [ 7759.295473] Lustre: server umount lustre-MDT0000 complete [ 7762.598710] LustreError: 235751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807185 with bad export cookie 17195136958457025447 [ 7762.600142] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.608982] LustreError: 235751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7762.948395] Lustre: server umount lustre-MDT0001 complete [ 7777.095817] Lustre: server umount lustre-OST0000 complete [ 7779.937157] Lustre: 235034:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786807186/real 1786807186] req@ffff8ccb6e460700 x1873603019760768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786807202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7779.961284] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.974480] Lustre: Skipped 1 previous similar message [ 7781.774555] Lustre: server umount lustre-OST0001 complete [ 7810.384484] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7821.050395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.750790] LustreError: 241187:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7821.851361] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.916268] LustreError: 241188:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.565423] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7832.040120] LustreError: 241187:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.157073] LustreError: 241188:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.400025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.906280] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.564939] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7846.067791] Lustre: 242333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7853.411554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.113480] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7862.060130] LustreError: 242686:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.166614] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 7869.100523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7869.416123] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7869.420839] Lustre: Skipped 1 previous similar message [ 7876.194134] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7884.858459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7889.335817] Lustre: 244201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7904.225748] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7904.234892] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.242765] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.076332] Lustre: server umount mylustre-MDT0000 complete [ 7910.378811] LustreError: 241187:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7910.402464] LustreError: 241187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7910.714125] LustreError: 241932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807333 with bad export cookie 17195136958457028254 [ 7910.717964] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.730378] LustreError: 241932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7911.107312] Lustre: server umount mylustre-MDT0001 complete [ 7925.965625] Lustre: server umount mylustre-OST0000 complete [ 7939.551987] Lustre: server umount mylustre-OST0001 complete [ 7966.204226] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7977.448509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.836136] LustreError: 246510:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7977.923227] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.403443] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7990.078868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.181394] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7996.463731] Lustre: 247650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8002.775447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8004.099992] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49792) [ 8010.040147] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8017.740691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.037947] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8023.656911] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8030.293437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8033.194849] Lustre: 249519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8041.440762] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8041.450682] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.462413] Lustre: Skipped 3 previous similar messages [ 8041.469180] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.476542] Lustre: Skipped 4 previous similar messages [ 8045.479296] Lustre: server umount tfs-MDT0000 complete [ 8048.613585] LustreError: 246511:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.643034] LustreError: 246511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8048.956610] LustreError: 246486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807471 with bad export cookie 17195136958457029913 [ 8048.959142] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8048.966927] LustreError: 246486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8049.299984] Lustre: server umount tfs-MDT0001 complete [ 8063.034654] Lustre: server umount tfs-OST0000 complete [ 8076.661957] Lustre: server umount tfs-OST0001 complete [ 8096.603028] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8105.007674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.381741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.391564] Lustre: Skipped 3 previous similar messages [ 8108.581440] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8115.678297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8119.972867] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8122.811931] Lustre: 252912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8129.518892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8130.885109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49824) [ 8135.221853] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8142.409079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8147.963985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8148.245719] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8155.070861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8158.427861] Lustre: 254784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8164.894987] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:26:25 (1786807585) [ 8182.756669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8182.765442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.782861] Lustre: Skipped 3 previous similar messages [ 8182.787062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.792000] Lustre: Skipped 4 previous similar messages [ 8186.327818] Lustre: server umount lustre-MDT0000 complete [ 8188.899478] LustreError: 251767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.906874] LustreError: 251767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8189.840836] LustreError: 252514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807612 with bad export cookie 17195136958457031579 [ 8189.847449] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.852846] LustreError: 252514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8194.017176] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.022259] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8194.043108] Lustre: Skipped 4 previous similar messages [ 8194.055813] Lustre: Skipped 5 previous similar messages [ 8204.256384] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.731863] Lustre: server umount lustre-MDT0001 complete [ 8223.201586] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8223.343265] Lustre: server umount lustre-OST0000 complete [ 8233.591056] Lustre: server umount lustre-OST0001 complete [ 8240.840943] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 8255.189541] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8303.704208] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8303.704208] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8303.704208] [ 8303.800796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8304.113713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8304.161458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8304.425381] Lustre: lustre-MDT0000: new disk, initializing [ 8304.616677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8304.631691] Lustre: Skipped 3 previous similar messages [ 8304.694990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8312.705716] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8328.762781] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8328.762781] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8328.762781] [ 8328.835644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8328.985497] Lustre: 257973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8329.044106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8329.047781] Lustre: Skipped 1 previous similar message [ 8329.130509] Lustre: lustre-MDT0001: new disk, initializing [ 8329.296201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8329.313237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8333.728179] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8345.101503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.567653] Lustre: lustre-OST0000: new disk, initializing [ 8345.574105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8345.582350] Lustre: 259558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8347.457244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8347.465560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8347.556291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8352.195578] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8363.013491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.341248] Lustre: lustre-OST0001: new disk, initializing [ 8363.353934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8363.364447] Lustre: 260430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8364.940232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8364.955197] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8365.061389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8370.821798] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8382.777222] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:30:03 (1786807803) [ 8396.134779] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:30:16 (1786807816) [ 8411.122924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.136829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.139110] Lustre: Skipped 1 previous similar message [ 8411.144120] Lustre: Skipped 4 previous similar messages [ 8417.121771] Lustre: server umount lustre-OST0000 complete [ 8426.480747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8428.363079] Lustre: server umount lustre-MDT0000 complete [ 8432.705488] LustreError: 257964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786807855 with bad export cookie 17195136958457033140 [ 8432.714496] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8432.727271] LustreError: 257964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8439.473506] Lustre: server umount lustre-MDT0001 complete [ 8451.001721] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8468.829023] Lustre: server umount lustre-OST0001 complete [ 8474.924919] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 8490.277407] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8533.538865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8533.810649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8533.846179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8533.996512] Lustre: lustre-MDT0000: new disk, initializing [ 8534.078801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8539.421340] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8549.937522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.063546] Lustre: 264986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8550.127576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8550.131763] Lustre: Skipped 1 previous similar message [ 8550.208796] Lustre: lustre-MDT0001: new disk, initializing [ 8550.340183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8550.359698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8555.113404] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8565.188294] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8572.057300] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8580.454618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.811111] Lustre: 266956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8580.938882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.943779] Lustre: Skipped 5 previous similar messages [ 8582.680032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8582.685702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8582.805426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8589.265344] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8599.219675] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8608.227280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8608.233449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.250742] Lustre: Skipped 4 previous similar messages [ 8608.256682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8608.261685] Lustre: Skipped 8 previous similar messages [ 8613.351367] Lustre: server umount lustre-OST0000 complete [ 8623.726761] Lustre: server umount lustre-MDT0000 complete [ 8624.096895] LustreError: 265866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.124978] LustreError: 265866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8627.757274] LustreError: 264981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808050 with bad export cookie 17195136958457034610 [ 8627.764864] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8627.781616] LustreError: 264981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8628.374531] Lustre: server umount lustre-MDT0001 complete [ 8640.903294] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8645.093524] Key type lgssc unregistered [ 8645.612934] LNet: 268717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8645.619602] LNetError: 268717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8645.661385] LNet: Removed LNI 192.168.204.111@tcp [ 8646.925473] Key type .llcrypt unregistered [ 8646.934377] Key type ._llcrypt unregistered [ 8673.674992] Key type ._llcrypt registered [ 8673.679263] Key type .llcrypt registered [ 8673.756420] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8674.417154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8674.432430] alg: No test for adler32 (adler32-zlib) [ 8675.576517] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 8675.797153] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 8677.447130] Key type lgssc registered [ 8678.670930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8686.294864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8686.320131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.651104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.729319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8697.830681] LustreError: 270182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.851463] LustreError: 270182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8700.096034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.445972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8704.897811] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8712.038257] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8717.056480] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8724.285985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8724.571188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.951568] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8732.794108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8739.901893] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8740.993628] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8743.396903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8743.398932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8743.414958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8743.420599] Lustre: Skipped 1 previous similar message [ 8748.518332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8748.529774] Lustre: Skipped 1 previous similar message [ 8749.395492] Lustre: server umount lustre-OST0000 complete [ 8753.635620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.643569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8753.657733] Lustre: Skipped 2 previous similar messages [ 8753.681432] Lustre: Skipped 1 previous similar message [ 8758.757165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8758.769648] Lustre: Skipped 1 previous similar message [ 8759.640695] Lustre: server umount lustre-MDT0000 complete [ 8763.032517] LustreError: 272713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808185 with bad export cookie 5099753692483732905 [ 8763.035691] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8763.056212] LustreError: 272713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8763.442281] Lustre: server umount lustre-MDT0001 complete [ 8774.183438] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8776.913382] Key type lgssc unregistered [ 8777.179635] LNet: 273769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8777.188210] LNetError: 273769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8777.222806] LNet: Removed LNI 192.168.204.111@tcp [ 8778.001207] Key type .llcrypt unregistered [ 8778.010471] Key type ._llcrypt unregistered [ 8794.514883] Key type ._llcrypt registered [ 8794.518271] Key type .llcrypt registered [ 8794.618679] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8795.799555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8795.907565] alg: No test for adler32 (adler32-zlib) [ 8796.941702] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 8797.162932] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 8798.841599] Key type lgssc registered [ 8799.886588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8809.404590] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8809.424602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8824.992455] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8830.111527] LustreError: 274962:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 8856.799249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.065409] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8877.382610] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8879.742768] Lustre: server umount lustre-OST0000 complete [ 8896.735631] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8899.575605] Key type lgssc unregistered [ 8899.988801] LNet: 276595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8900.002221] LNetError: 276595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8900.064690] LNet: Removed LNI 192.168.204.111@tcp [ 8900.903077] Key type .llcrypt unregistered [ 8900.905844] Key type ._llcrypt unregistered [ 8951.504210] Key type ._llcrypt registered [ 8951.507529] Key type .llcrypt registered [ 8951.659864] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8952.748979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8952.824980] alg: No test for adler32 (adler32-zlib) [ 8953.924369] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 8954.223352] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 8955.951452] Key type lgssc registered [ 8957.466634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8974.941740] loop0: detected capacity change from 0 to 204800000 [ 8975.175223] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8975.208530] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8975.586825] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8976.435850] loop0: detected capacity change from 0 to 204800000 [ 8976.568334] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8976.588919] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8976.811406] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8978.110439] loop0: detected capacity change from 0 to 204800000 [ 8978.376496] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8978.391522] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8978.470373] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8979.526198] loop0: detected capacity change from 0 to 204800000 [ 8979.731405] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8981.030146] loop0: detected capacity change from 0 to 204800000 [ 8981.049821] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8985.049462] loop0: detected capacity change from 0 to 204800000 [ 8985.066559] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8989.780618] loop0: detected capacity change from 0 to 204800000 [ 8989.820211] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8994.635164] loop0: detected capacity change from 0 to 204800000 [ 8994.672471] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8999.147549] loop0: detected capacity change from 0 to 204800000 [ 8999.226808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8999.279840] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8999.386504] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8999.400281] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8999.420199] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8999.429257] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8999.439220] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8999.451576] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8999.465488] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8999.484598] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8999.495198] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8999.510144] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8999.520431] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8999.532582] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8999.540878] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8999.557625] Lustre: 280014:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9006.045661] Lustre: server umount MGS complete [ 9007.431860] loop0: detected capacity change from 0 to 204800000 [ 9007.612202] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.476069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.547764] Lustre: lustre-MDT0000: Aborting client recovery [ 9009.674785] loop1: detected capacity change from 0 to 204800000 [ 9009.825130] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.964847] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9009.999210] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9010.012278] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9010.041841] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9010.059314] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9010.084628] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9010.122481] Lustre: 280447:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9010.169051] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9010.392417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9010.437924] Lustre: lustre-MDT0001: Aborting client recovery [ 9011.361033] loop2: detected capacity change from 0 to 204800000 [ 9011.495839] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9011.600768] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9011.618514] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9011.641242] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9011.663668] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9011.685636] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9011.708520] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9011.734754] Lustre: 280528:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9011.856436] Lustre: Found index 0 for lustre-OST0000, updating log [ 9011.880545] Lustre: 280546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9011.956825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.973365] Lustre: lustre-OST0000: Aborting recovery [ 9012.897572] loop3: detected capacity change from 0 to 204800000 [ 9013.062428] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.157373] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9013.171926] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9013.192929] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9013.211844] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9013.226393] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9013.234443] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9013.242424] Lustre: 280611:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9013.256939] Lustre: Found index 1 for lustre-OST0001, updating log [ 9013.281428] Lustre: 280616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9013.348651] Lustre: lustre-OST0001: Aborting recovery [ 9018.358320] LustreError: 280391:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9019.366210] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9019.368464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9019.457232] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9019.484269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9020.393259] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9020.393616] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9023.271413] LustreError: 280358:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 6345us: rc = -2 [ 9023.825366] LustreError: 280358:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 440us: rc = -2 [ 9023.857717] LustreError: 280358:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 9040.868595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.877824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9040.903260] Lustre: Skipped 2 previous similar messages [ 9040.905031] Lustre: Skipped 1 previous similar message [ 9044.716398] Lustre: server umount lustre-MDT0000 complete [ 9045.614983] LustreError: 280340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808468 with bad export cookie 2813965839034227746 [ 9045.623791] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9045.631419] LustreError: 280340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9045.983799] LustreError: 280623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.985653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9045.992866] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9045.992877] Lustre: Skipped 2 previous similar messages [ 9046.007190] LustreError: 280623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9046.016825] Lustre: Skipped 2 previous similar messages [ 9051.106608] LustreError: 280357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9051.108991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9051.168893] Lustre: Skipped 2 previous similar messages [ 9052.053713] Lustre: server umount lustre-MDT0001 complete [ 9059.080193] Lustre: server umount lustre-OST0000 complete [ 9066.341518] Lustre: server umount lustre-OST0001 complete [ 9075.498306] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:41:35 (1786808495) [ 9108.282979] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 9118.799502] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9156.353374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9156.574794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9156.610228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9156.714574] Lustre: lustre-MDT0000: new disk, initializing [ 9156.794089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.800694] Lustre: Skipped 1 previous similar message [ 9156.813731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9161.417754] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9171.200322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.333986] Lustre: 284686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9171.392499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9171.396645] Lustre: Skipped 1 previous similar message [ 9171.462624] Lustre: lustre-MDT0001: new disk, initializing [ 9171.558725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.597690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9171.609196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9176.240832] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9186.431545] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9193.780024] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9202.187986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9202.714769] Lustre: lustre-OST0000: new disk, initializing [ 9202.723499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9202.734792] Lustre: 286653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9202.887957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.307553] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9212.528387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9212.542295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9212.687302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9224.939634] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9230.893408] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9243.637985] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9251.226190] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9299.943041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.944452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9299.946878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9299.958420] Lustre: Skipped 1 previous similar message [ 9299.969374] LustreError: Skipped 1 previous similar message [ 9303.546862] Lustre: server umount lustre-OST0000 complete [ 9309.669057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9309.678067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.703625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9309.708418] Lustre: Skipped 1 previous similar message [ 9314.081141] Lustre: server umount lustre-MDT0000 complete [ 9315.296643] LustreError: 284697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.328904] LustreError: 284697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9317.873153] LustreError: 284677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808740 with bad export cookie 2813965839034230497 [ 9317.878309] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.888498] LustreError: 284677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9318.193304] Lustre: server umount lustre-MDT0001 complete [ 9326.599237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.407307] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9336.262723] Lustre: server umount MGS complete [ 9348.489369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.010905] LustreError: 290637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.036178] LustreError: 290650:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9349.129250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.913078] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9354.210404] LustreError: 290638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.431257] LustreError: 290637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.450722] LustreError: 290637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9363.459352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.693384] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9376.775268] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9382.009739] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9387.681455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.981035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9387.990557] Lustre: Skipped 1 previous similar message [ 9393.807287] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9400.314509] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9434.081956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9434.086369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9434.101122] Lustre: Skipped 1 previous similar message [ 9434.110614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.122997] Lustre: Skipped 2 previous similar messages [ 9444.319128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9444.671809] Lustre: server umount lustre-OST0000 complete [ 9450.979396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9450.991546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9451.008898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9451.019547] Lustre: Skipped 4 previous similar messages [ 9455.931547] Lustre: server umount lustre-MDT0000 complete [ 9459.696937] LustreError: 290638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.711569] LustreError: 290638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9460.746785] LustreError: 290619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786808883 with bad export cookie 2813965839034232317 [ 9460.767222] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9460.775074] LustreError: 290619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9461.341117] Lustre: server umount lustre-MDT0001 complete [ 9476.389027] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 9481.241867] Key type lgssc unregistered [ 9481.469773] LNet: 294584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9481.474706] LNetError: 294584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9481.502198] LNet: Removed LNI 192.168.204.111@tcp [ 9482.341953] Key type .llcrypt unregistered [ 9482.344062] Key type ._llcrypt unregistered [ 9520.174620] Key type ._llcrypt registered [ 9520.176912] Key type .llcrypt registered [ 9520.290069] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 9534.337335] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9535.339424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9535.386965] alg: No test for adler32 (adler32-zlib) [ 9536.443186] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 9536.635054] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 9538.295777] Key type lgssc registered [ 9539.248065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9580.117987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9580.145301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.330182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9581.352736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9581.413535] Lustre: lustre-MDT0000: new disk, initializing [ 9581.484428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.511228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9586.944996] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9599.257780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9599.386858] Lustre: 298543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9599.430797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9599.433745] Lustre: Skipped 1 previous similar message [ 9599.492581] Lustre: lustre-MDT0001: new disk, initializing [ 9599.621625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9599.688488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9599.704733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9604.028353] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9613.919723] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9620.513591] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9628.694059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9629.058330] Lustre: lustre-OST0000: new disk, initializing [ 9629.061756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9629.076273] Lustre: 300511:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9629.218347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.870860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9635.884643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9635.988306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9636.348547] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9644.920872] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9652.076534] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9663.330426] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9670.904305] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9722.854787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.856798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.875286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.884962] Lustre: Skipped 1 previous similar message [ 9725.766539] Lustre: server umount lustre-OST0000 complete [ 9732.581624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9732.585273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.596625] Lustre: Skipped 1 previous similar message [ 9732.599811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9738.211062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9738.218166] Lustre: Skipped 3 previous similar messages [ 9742.304970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.313106] Lustre: Skipped 1 previous similar message [ 9743.327200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9743.615652] Lustre: server umount lustre-MDT0000 complete [ 9747.618247] LustreError: 298536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809170 with bad export cookie 5041073661215852641 [ 9747.637792] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.646456] LustreError: 298536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9748.321813] Lustre: server umount lustre-MDT0001 complete [ 9760.394676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.589443] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9774.612602] Lustre: server umount MGS complete [ 9787.370217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9787.872956] LustreError: 304427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9787.888403] LustreError: 304427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9787.900697] LustreError: 304440:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9787.967139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.393913] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9792.998199] LustreError: 304428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.093870] LustreError: 304427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.991615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9802.208618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9802.411265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9806.558134] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9813.746125] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9819.470450] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9826.173722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9826.529789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.752182] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9840.140125] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9868.780810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9868.787425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.802405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9868.803599] Lustre: Skipped 2 previous similar messages [ 9868.824503] Lustre: Skipped 1 previous similar message [ 9873.348626] Lustre: server umount lustre-OST0000 complete [ 9879.009778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.068029] Lustre: Skipped 1 previous similar message [ 9888.228106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9888.232419] Lustre: Skipped 5 previous similar messages [ 9892.319617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9892.495141] Lustre: server umount lustre-MDT0000 complete [ 9893.352093] LustreError: 304422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.380823] LustreError: 304422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9895.718444] LustreError: 304408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786809318 with bad export cookie 5041073661215854496 [ 9895.719560] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9895.730087] LustreError: 304408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9896.001983] Lustre: server umount lustre-MDT0001 complete [ 9908.211394] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 9910.684610] Key type lgssc unregistered [ 9910.935136] LNet: 308373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9910.942590] LNetError: 308373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9910.979213] LNet: Removed LNI 192.168.204.111@tcp [ 9912.097393] Key type .llcrypt unregistered [ 9912.108373] Key type ._llcrypt unregistered [ 9934.113732] Key type ._llcrypt registered [ 9934.116984] Key type .llcrypt registered [ 9934.228204] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 9948.224765] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9949.368413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9949.414934] alg: No test for adler32 (adler32-zlib) [ 9950.889870] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [ 9951.411261] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 9953.095918] Key type lgssc registered [ 9955.457038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9982.075820] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9996.278337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9996.292543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.477258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9997.500527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9997.599279] Lustre: lustre-MDT0000: new disk, initializing [ 9997.687855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.720763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10001.423923] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10011.417632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10011.694788] Lustre: lustre-OST0000: new disk, initializing [10011.704171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10011.719811] Lustre: Skipped 1 previous similar message [10011.737620] Lustre: 312417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10011.856942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10016.855047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10016.868942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10016.906526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10017.804917] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10969.057189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.074959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.216030] Lustre: server umount lustre-MDT0000 complete [10975.685407] LustreError: 312416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810398 with bad export cookie 5672214761939199943 [10975.690775] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10975.706243] LustreError: 312416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10975.908424] Lustre: server umount lustre-OST0000 complete [11005.657786] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [11020.388299] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [11072.276217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11072.618861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11072.681380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11072.805174] Lustre: lustre-MDT0000: new disk, initializing [11072.932748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11072.948347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11077.913281] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11089.922921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11090.105568] Lustre: 316776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11090.237612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11090.245537] Lustre: Skipped 1 previous similar message [11090.394698] Lustre: lustre-MDT0001: new disk, initializing [11090.623036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11090.713949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11090.730106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11096.387769] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11107.071380] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11114.229588] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11122.738604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11123.005240] Lustre: lustre-OST0000: new disk, initializing [11123.008453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11123.013475] Lustre: 318745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11123.115244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11129.806864] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11130.463882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11130.497389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11130.565928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11140.061245] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11146.066023] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11146.375496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11151.782183] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11152.073935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11156.464568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11156.472530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.478247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11156.490421] Lustre: Skipped 1 previous similar message [11156.507965] Lustre: Skipped 1 previous similar message [11161.581357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11161.591446] Lustre: Skipped 1 previous similar message [11166.692680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11166.701946] Lustre: Skipped 1 previous similar message [11168.224762] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11168.629468] Lustre: server umount lustre-OST0000 complete [11176.932912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.965230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11176.970024] Lustre: Skipped 1 previous similar message [11179.895831] LustreError: 320287:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000081c9f529) refcount nonzero (2) after lock cleanup; forcing cleanup. [11180.283408] Lustre: server umount lustre-MDT0000 complete [11185.142651] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810608 with bad export cookie 5672214761942140762 [11185.145597] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11185.169603] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11185.849276] Lustre: server umount lustre-MDT0001 complete [11190.745307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11193.318578] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:16:53 (1786810613) [11202.095106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.792451] LustreError: 321061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.822072] LustreError: 321061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11202.943576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11208.165163] LustreError: 321062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.557416] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11212.270111] LustreError: 321061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11217.377292] LustreError: 321062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11219.741446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11220.190501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11225.596870] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11235.358420] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11241.891436] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11249.904473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.185714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11257.874795] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11267.202163] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11274.534072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11274.741779] Lustre: lustre-OST0001: new disk, initializing [11274.749240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11274.755787] Lustre: 323900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11274.901952] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11281.481773] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11282.011203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11282.028408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11282.101519] LustreError: 324287:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11283.176051] LustreError: 324280:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11283.184546] LustreError: 324280:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11284.197290] LustreError: 324287:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11286.243281] LustreError: 324287:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11286.252853] LustreError: 324287:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11290.405763] LustreError: 324280:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11290.414638] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11290.417788] LustreError: 324280:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [11290.656442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11296.687664] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fb2857eb000.ost_server_uuid 50 [11298.531270] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9fb2857eb000.ost_server_uuid in FULL state after 0 sec [11298.595752] LustreError: 324287:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11298.610542] LustreError: 324287:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11304.735293] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fb2857eb000.ost_server_uuid 50 [11306.322920] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fb2857eb000.ost_server_uuid in FULL state after 0 sec [11307.962277] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11322.251995] Lustre: lustre-OST0001: Not available for connect from 192.168.204.11@tcp (stopping) [11323.364153] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11323.366736] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11323.383059] Lustre: Skipped 1 previous similar message [11323.401231] LustreError: Skipped 1 previous similar message [11327.612966] Lustre: server umount lustre-OST0001 complete [11328.501831] LustreError: 324289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11328.517441] LustreError: 324289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11333.613743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.622408] Lustre: Skipped 1 previous similar message [11338.208635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11338.727229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11338.735997] Lustre: Skipped 5 previous similar messages [11338.737722] LustreError: 322881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.756817] LustreError: 322881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11339.813638] Lustre: server umount lustre-OST0000 complete [11348.449451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11348.464798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11348.500374] Lustre: Skipped 1 previous similar message [11359.200978] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11359.494186] Lustre: server umount lustre-MDT0000 complete [11363.296404] LustreError: 321062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11363.321862] LustreError: 321062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11364.255733] LustreError: 321042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786810787 with bad export cookie 5672214761942141644 [11364.264134] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11364.267871] LustreError: 321042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11364.682446] Lustre: server umount lustre-MDT0001 complete [11377.270723] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [11380.569348] Key type lgssc unregistered [11381.033230] LNet: 326592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11381.046926] LNetError: 326592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11381.071466] LNet: Removed LNI 192.168.204.111@tcp [11382.423450] Key type .llcrypt unregistered [11382.425753] Key type ._llcrypt unregistered [11413.647136] Key type ._llcrypt registered [11413.648652] Key type .llcrypt registered [11413.809462] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [11415.261735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.331590] alg: No test for adler32 (adler32-zlib) [11416.612768] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [11416.812854] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [11418.463312] Key type lgssc registered [11419.373170] Lustre: Echo OBD driver; http://www.lustre.org/ [11427.049566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11427.065662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11428.433797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11432.693946] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11438.565302] LustreError: 328011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.588550] LustreError: 328011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11439.908520] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11443.693092] LustreError: 328010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.264628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.734874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11452.188054] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11462.082991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11462.534565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11470.327172] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11472.814611] LustreError: 329688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.854434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11479.014445] LustreError: 330185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.038233] LustreError: 330185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11479.188481] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11484.141129] LustreError: 329688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.153217] LustreError: 329688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11486.094727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11486.317146] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11491.866875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11492.233961] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11499.751791] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11508.681908] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11508.979499] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11614.689413] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11614.692124] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11614.705039] LustreError: Skipped 1 previous similar message [11614.716657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11614.733505] Lustre: Skipped 1 previous similar message [11619.815788] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11620.093647] Lustre: server umount lustre-OST0001 complete [11624.937475] LustreError: 330167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.947361] LustreError: 330167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11626.467669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11626.474930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11626.494576] Lustre: Skipped 1 previous similar message [11626.498198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11626.504083] Lustre: Skipped 1 previous similar message [11627.490937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11627.495745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.050852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11630.068604] Lustre: Skipped 2 previous similar messages [11632.579934] Lustre: server umount lustre-OST0000 complete [11637.217437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11637.230809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11637.248545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11645.409270] LustreError: 330578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.420825] LustreError: 330578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11645.450714] Lustre: server umount lustre-MDT0000 complete [11649.434605] LustreError: 327991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811072 with bad export cookie 12966952104485817308 [11649.436089] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11649.453470] LustreError: 327991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11649.825929] Lustre: server umount lustre-MDT0001 complete [11663.571992] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [11667.918812] Key type lgssc unregistered [11668.329542] LNet: 332955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11668.343684] LNetError: 332955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11668.363921] LNet: Removed LNI 192.168.204.111@tcp [11669.750516] Key type .llcrypt unregistered [11669.757792] Key type ._llcrypt unregistered [11720.742753] Key type ._llcrypt registered [11720.745459] Key type .llcrypt registered [11720.849538] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [11721.850747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11721.948772] alg: No test for adler32 (adler32-zlib) [11723.112536] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [11723.313570] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [11724.967200] Key type lgssc registered [11726.039635] Lustre: Echo OBD driver; http://www.lustre.org/ [11734.970841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11734.984969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11736.424189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11740.219230] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11746.790491] LustreError: 335262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.813105] LustreError: 335262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11747.870948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11748.308680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11752.484411] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11755.136942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11761.471467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11761.672075] LustreError: 336751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.003574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.274517] LustreError: 336751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.289138] LustreError: 336751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11767.291719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11768.318260] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11772.394556] LustreError: 336752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.105491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11776.307399] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11781.624828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11783.215390] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11792.434081] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11799.847669] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11812.328392] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11812.332925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11812.364393] Lustre: Skipped 2 previous similar messages [11812.370650] Lustre: Skipped 1 previous similar message [11817.440187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11817.458061] Lustre: Skipped 5 previous similar messages [11818.388262] Lustre: server umount lustre-MDT0000 complete [11822.192405] LustreError: 335242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811245 with bad export cookie 5057150937409943722 [11822.195535] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11822.211514] LustreError: 335242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11822.561183] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.562497] LustreError: 335258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.577676] Lustre: Skipped 2 previous similar messages [11822.623534] LustreError: 335258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11822.823122] Lustre: server umount lustre-MDT0001 complete [11829.056833] Lustre: server umount lustre-OST0000 complete [11832.744883] Lustre: server umount lustre-OST0001 complete [11850.039756] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [11862.493861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11863.086449] LustreError: 340521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.154261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11867.574119] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11876.868373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11877.471712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11883.865581] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11887.469412] Lustre: 341662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11895.800973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11896.351658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11902.502552] LustreError: 342017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.532501] LustreError: 342017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11903.005769] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11908.654856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11911.867148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11917.331256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11918.659296] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11926.975677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11931.279580] Lustre: 343533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11937.766901] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11937.775130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11937.790754] Lustre: Skipped 2 previous similar messages [11937.809965] Lustre: Skipped 3 previous similar messages [11941.992305] Lustre: server umount lustre-MDT0000 complete [11942.881409] LustreError: 343397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.915346] LustreError: 343397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11947.054266] LustreError: 340502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811370 with bad export cookie 5057150937409945423 [11947.071499] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11947.086304] LustreError: 340502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11947.701219] Lustre: server umount lustre-MDT0001 complete [11963.620069] Lustre: 334524:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786811370/real 1786811370] req@ffff8ccb7d9f1c00 x1873607284342784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786811386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11963.660670] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.677764] Lustre: Skipped 1 previous similar message [11963.868293] Lustre: server umount lustre-OST0000 complete [11967.675494] Lustre: server umount lustre-OST0001 complete [11971.455263] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11973.317889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11975.261546] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 12:29:55 (1786811395) [12002.385668] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [12011.734885] loop0: detected capacity change from 0 to 1099511627776 [12012.365509] SGI XFS with ACLs, security attributes, quota, no debug enabled [12012.381759] *** VALIDATE xfs *** [12012.396521] XFS (loop0): Mounting V5 Filesystem [12012.416519] XFS (loop0): Ending clean mount [12013.630487] loop1: detected capacity change from 0 to 18691697672192 [12128.279776] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [12138.376369] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [12176.969769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12177.290757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12177.316934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12177.378256] Lustre: lustre-MDT0000: new disk, initializing [12177.449373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12177.458948] Lustre: Skipped 1 previous similar message [12177.479140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12182.444501] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12195.138315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12195.250572] Lustre: 348669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12195.311246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12195.316326] Lustre: Skipped 1 previous similar message [12195.385855] Lustre: lustre-MDT0001: new disk, initializing [12195.516224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12195.539757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12200.984953] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12211.195285] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12217.647734] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12225.278534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12225.644895] Lustre: lustre-OST0000: new disk, initializing [12225.654728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12225.668730] Lustre: 350637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12227.372264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12227.384392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12227.476092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12232.789542] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12244.106046] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12249.598693] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12249.892358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12255.547814] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12255.849233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12258.274459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12258.276273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.278857] LustreError: Skipped 1 previous similar message [12258.284909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12258.289216] Lustre: Skipped 1 previous similar message [12263.399879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12263.409655] Lustre: Skipped 1 previous similar message [12264.679503] Lustre: server umount lustre-OST0000 complete [12272.610741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12272.624673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12272.648031] Lustre: Skipped 1 previous similar message [12272.665744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12283.359217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12283.875646] LustreError: 348675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.906330] LustreError: 348675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12284.067192] Lustre: server umount lustre-MDT0000 complete [12288.649022] LustreError: 348662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811711 with bad export cookie 5057150937409946970 [12288.656177] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12289.183808] Lustre: server umount lustre-MDT0001 complete [12299.300505] XFS (loop0): Unmounting Filesystem [12304.532383] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 12:35:24 (1786811724) [12312.518119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.945680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12312.952748] Lustre: Skipped 2 previous similar messages [12317.039246] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12326.611370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12331.383800] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12339.967354] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12347.243729] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12354.813389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12360.943795] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12369.389395] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12382.688778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12382.694389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.705347] Lustre: Skipped 1 previous similar message [12382.710371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12382.715855] Lustre: Skipped 4 previous similar messages [12388.084777] Lustre: server umount lustre-OST0000 complete [12397.999932] Lustre: server umount lustre-MDT0000 complete [12402.462815] LustreError: 353182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786811825 with bad export cookie 5057150937409947852 [12402.491415] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12402.492380] LustreError: 353182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12402.903061] Lustre: server umount lustre-MDT0001 complete [12419.550809] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [12423.382961] Key type lgssc unregistered [12423.810620] LNet: 356830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12423.819626] LNetError: 356830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12423.836955] LNet: Removed LNI 192.168.204.111@tcp [12425.065690] Key type .llcrypt unregistered [12425.068181] Key type ._llcrypt unregistered [12457.408603] Key type ._llcrypt registered [12457.410824] Key type .llcrypt registered [12457.511157] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [12458.638113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12458.806371] alg: No test for adler32 (adler32-zlib) [12459.869369] Lustre: Lustre: Build Version: 2.17.56_1_gfa464be [12460.110980] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [12461.879228] Key type lgssc registered [12463.140154] Lustre: Echo OBD driver; http://www.lustre.org/ [12471.462598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12471.488651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12473.067404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12478.336108] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12482.535684] LustreError: 358401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12482.554433] LustreError: 358401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12487.660949] LustreError: 358402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12487.807655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12488.180240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12493.020320] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12501.077672] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12507.411141] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12514.357994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12514.662529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12522.259176] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12530.592375] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12542.849196] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [12543.460700] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.478561] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12543.486103] Lustre: Skipped 1 previous similar message [12546.681872] Lustre: server umount lustre-MDT0001 complete [12547.944038] LustreError: 359400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12553.055750] LustreError: 359400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12553.080559] LustreError: 359400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12556.419484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12556.547974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12556.554409] Lustre: Found index 1 for lustre-MDT0001, updating log [12556.616629] Lustre: 358391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12556.993965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12561.505540] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12562.414180] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12562.434486] LustreError: 361911:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12562.444727] LustreError: 361911:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12562.457275] LustreError: 358393:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12562.544515] LustreError: 361918:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -17 [12562.559283] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12562.559283] [12572.641817] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12572.650613] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12572.666965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12572.682471] Lustre: Skipped 2 previous similar messages [12572.702762] Lustre: Skipped 1 previous similar message [12577.763479] LustreError: 358398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12577.806423] LustreError: 358398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12577.901042] Lustre: server umount lustre-MDT0001 complete [12588.002492] LustreError: 358396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12588.055897] LustreError: 358396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12593.528036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12593.621770] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12593.627504] Lustre: Found index 1 for lustre-MDT0001, updating log [12593.712145] Lustre: 358391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12594.064794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12599.278203] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12599.360366] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12599.859244] LustreError: 362937:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12599.876260] LustreError: 362937:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12599.895038] LustreError: 358393:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12599.913991] LustreError: 358393:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12600.050578] LustreError: 362948:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12600.070731] LustreError: 362948:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12600.082388] LustreError: 362948:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -17 [12600.097845] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.111@tcp [12600.097845] [12911.499815] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [12911.517338] Lustre: Skipped 1 previous similar message [12911.587166] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.589191] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12913.493980] Lustre: server umount lustre-MDT0001 complete [12916.605618] LustreError: 358396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.637603] LustreError: 358396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12928.728583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12928.968794] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12928.974245] Lustre: Found index 1 for lustre-MDT0001, updating log [12929.015058] Lustre: 358391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12929.393983] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12933.758152] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [12934.648576] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12934.662199] LustreError: 371188:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12934.670524] LustreError: 371188:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12934.680069] LustreError: 358393:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12934.691836] LustreError: 358393:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12934.768971] LustreError: 371203:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12934.782987] LustreError: 371203:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -114 [12934.797467] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12934.797467] [13262.306719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.315735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13262.319909] Lustre: Skipped 3 previous similar messages [13262.345816] Lustre: Skipped 4 previous similar messages [13264.630154] Lustre: server umount lustre-MDT0000 complete [13267.426389] LustreError: 359400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13267.460357] LustreError: 359400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [13268.948904] LustreError: 358382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812691 with bad export cookie 9021891161150509983 [13268.951372] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13268.957106] LustreError: 358382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13272.546657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13272.562865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13283.295213] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13283.790828] Lustre: server umount lustre-MDT0001 complete [13288.425522] Lustre: server umount lustre-OST0000 complete [13298.856336] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [13313.369437] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [13354.261604] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:52:54 (1786812774) [13361.037381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13361.262494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13361.294435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13361.449619] Lustre: lustre-MDT0000: new disk, initializing [13361.578019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13361.594140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13366.168324] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [13380.440722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13380.607505] Lustre: 381887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13380.732251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13380.743256] Lustre: Skipped 1 previous similar message [13380.876929] Lustre: lustre-MDT0001: new disk, initializing [13381.001463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13381.071221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13381.098694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13387.075114] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [13400.031902] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13407.872647] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13415.838042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13416.069524] Lustre: lustre-OST0000: new disk, initializing [13416.073100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13416.080851] Lustre: 383857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13416.184196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13421.673688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13421.684977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13421.858689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13423.827985] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [13434.108862] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13447.649349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13447.653170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13447.660811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13447.660819] Lustre: Skipped 2 previous similar messages [13447.701254] Lustre: Skipped 1 previous similar message [13449.477531] Lustre: server umount lustre-MDT0000 complete [13452.770562] LustreError: 383449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13452.795749] LustreError: 383449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13453.383265] LustreError: 382681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786812876 with bad export cookie 9021891161150511936 [13453.393561] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13453.418721] LustreError: 382681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13454.017965] Lustre: server umount lustre-MDT0001 complete [13468.736610] Lustre: server umount lustre-OST0000 complete [13500.939955] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [13514.090700] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [13547.664410] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13294 sec ========= 12:56:07 (1786812967) [13549.499647] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:56:09 (1786812969) === [13553.336259] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:56:13 (1786812973) === [13597.722525] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [13600.788332] Key type lgssc unregistered [13601.184047] LNet: 390226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13601.201353] LNetError: 390226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13601.222504] LNet: Removed LNI 192.168.204.111@tcp [13602.212327] Key type .llcrypt unregistered [13602.213939] Key type ._llcrypt unregistered