[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492224094 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003326] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007856] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010224] LSM: Security Framework initializing [ 0.011066] Yama: becoming mindful. [ 0.012049] SELinux: Initializing. [ 0.013111] *** VALIDATE selinux *** [ 0.022100] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028035] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032103] *** VALIDATE tmpfs *** [ 0.034054] *** VALIDATE proc *** [ 0.035234] *** VALIDATE cgroup *** [ 0.036014] *** VALIDATE cgroup2 *** [ 0.037289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044357] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.047181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048719] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051017] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.056329] rcu: Hierarchical SRCU implementation. [ 0.058428] smp: Bringing up secondary CPUs ... [ 0.059678] x86: Booting SMP configuration: [ 0.060040] .... node #0, CPUs: #1 #2 #3 [ 0.065084] smp: Brought up 1 node, 4 CPUs [ 0.067016] smpboot: Max logical packages: 1 [ 0.068012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.142344] node 0 deferred pages initialised in 72ms [ 0.146145] devtmpfs: initialized [ 0.147655] x86/mm: Memory block size: 128MB [ 0.149272] gcov: version magic: 0x41383552 [ 0.152430] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159350] pinctrl core: initialized pinctrl subsystem [ 0.161183] [ 0.162011] ************************************************************* [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167014] ** ** [ 0.170013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172012] ** ** [ 0.175015] ** This means that this kernel is built to expose internal ** [ 0.177058] ** IOMMU data structures, which may compromise security on ** [ 0.180015] ** your system. ** [ 0.183015] ** ** [ 0.185013] ** If you see this message and you are not debugging the ** [ 0.188017] ** kernel, report this immediately to your vendor! ** [ 0.191015] ** ** [ 0.194017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196011] ************************************************************* [ 0.199816] NET: Registered protocol family 16 [ 0.201441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211039] cpuidle: using governor menu [ 0.212747] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216537] PCI: Using configuration type 1 for base access [ 0.218127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231118] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235157] cryptd: max_cpu_qlen set to 1000 [ 0.238208] ACPI: Added _OSI(Module Device) [ 0.241023] ACPI: Added _OSI(Processor Device) [ 0.243015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244077] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249281] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258307] ACPI: Interpreter enabled [ 0.260084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262020] ACPI: Using IOAPIC for interrupt routing [ 0.264321] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268745] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280071] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.282046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293370] acpiphp: Slot [2] registered [ 0.295214] acpiphp: Slot [5] registered [ 0.296171] acpiphp: Slot [6] registered [ 0.298153] acpiphp: Slot [7] registered [ 0.299169] acpiphp: Slot [8] registered [ 0.301129] acpiphp: Slot [9] registered [ 0.302151] acpiphp: Slot [10] registered [ 0.304148] acpiphp: Slot [3] registered [ 0.305131] acpiphp: Slot [4] registered [ 0.307152] acpiphp: Slot [11] registered [ 0.308124] acpiphp: Slot [12] registered [ 0.309122] acpiphp: Slot [13] registered [ 0.311244] acpiphp: Slot [14] registered [ 0.312119] acpiphp: Slot [15] registered [ 0.314121] acpiphp: Slot [16] registered [ 0.315142] acpiphp: Slot [17] registered [ 0.317132] acpiphp: Slot [18] registered [ 0.319139] acpiphp: Slot [19] registered [ 0.321121] acpiphp: Slot [20] registered [ 0.322115] acpiphp: Slot [21] registered [ 0.324119] acpiphp: Slot [22] registered [ 0.325116] acpiphp: Slot [23] registered [ 0.327139] acpiphp: Slot [24] registered [ 0.329097] acpiphp: Slot [25] registered [ 0.330133] acpiphp: Slot [26] registered [ 0.332113] acpiphp: Slot [27] registered [ 0.333097] acpiphp: Slot [28] registered [ 0.335214] acpiphp: Slot [29] registered [ 0.337178] acpiphp: Slot [30] registered [ 0.339101] acpiphp: Slot [31] registered [ 0.340076] PCI host bridge to bus 0000:00 [ 0.342023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.355032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361405] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365378] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375779] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.383072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.385023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.395666] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399148] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.402065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.410018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.429022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.436034] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.444024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.467018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.479257] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.520015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.533614] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.540017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.547019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.577219] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.583016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.590018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.605032] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.618397] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.623019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.628021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.643016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.654590] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.665023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.677022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.694027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.706075] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.709382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711512] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.715497] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.719321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.724145] iommu: Default domain type: Passthrough [ 0.726624] SCSI subsystem initialized [ 0.728151] ACPI: bus type USB registered [ 0.729172] usbcore: registered new interface driver usbfs [ 0.732160] usbcore: registered new interface driver hub [ 0.734111] usbcore: registered new device driver usb [ 0.735223] pps_core: LinuxPPS API ver. 1 registered [ 0.738017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.741100] PTP clock support registered [ 0.743246] EDAC MC: Ver: 3.0.0 [ 0.745176] PCI: Using ACPI for IRQ routing [ 0.746969] NetLabel: Initializing [ 0.748012] NetLabel: domain hash size = 128 [ 0.749019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751099] NetLabel: unlabeled traffic allowed by default [ 0.754075] vgaarb: loaded [ 0.756403] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.758024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765000] clocksource: Switched to clocksource kvm-clock [ 0.877684] VFS: Disk quotas dquot_6.6.0 [ 0.879290] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.882320] *** VALIDATE ramfs *** [ 0.883518] *** VALIDATE hugetlbfs *** [ 0.885198] pnp: PnP ACPI init [ 0.887904] pnp: PnP ACPI: found 6 devices [ 0.921989] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.926165] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.928211] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.929917] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.932138] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.934452] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.937313] NET: Registered protocol family 2 [ 0.940646] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.946165] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.949963] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.955781] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.959497] TCP: Hash tables configured (established 65536 bind 65536) [ 0.962913] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.966352] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.969398] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972159] NET: Registered protocol family 1 [ 0.977109] RPC: Registered named UNIX socket transport module. [ 0.980236] RPC: Registered udp transport module. [ 0.982097] RPC: Registered tcp transport module. [ 0.983696] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986368] NET: Registered protocol family 44 [ 0.988466] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990812] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.993462] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.996487] PCI: CLS 0 bytes, default 64 [ 0.998318] Unpacking initramfs... [ 2.741639] debug: unmapping init [mem 0xffff94b93cc54000-0xffff94b93ffbffff] [ 2.758380] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.763813] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.768187] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 6.066433] Initialise system trusted keyrings [ 6.071390] Key type blacklist registered [ 6.078330] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.110591] zbud: loaded [ 6.121680] *** VALIDATE nfs *** [ 6.123890] *** VALIDATE nfs4 *** [ 6.132694] pstore: using deflate compression [ 6.170684] Platform Keyring initialized [ 6.580537] NET: Registered protocol family 38 [ 6.589113] Key type asymmetric registered [ 6.595107] Asymmetric key parser 'x509' registered [ 6.600301] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.611354] io scheduler mq-deadline registered [ 6.613919] io scheduler kyber registered [ 6.619899] io scheduler bfq registered [ 6.627415] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.632549] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.637849] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.647839] ACPI: Power Button [PWRF] [ 6.666791] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.680745] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.710835] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.725413] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.783650] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.834355] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.897803] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.927588] Non-volatile memory driver v1.3 [ 6.930552] Linux agpgart interface v0.103 [ 7.006205] virtio_blk virtio1: [vda] 149456 512-byte logical blocks (76.5 MB/73.0 MiB) [ 7.017446] vda: detected capacity change from 0 to 76521472 [ 7.057134] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.062986] vdb: detected capacity change from 0 to 1073741824 [ 7.098608] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.104577] vdc: detected capacity change from 0 to 2621440000 [ 7.147730] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.155110] vdd: detected capacity change from 0 to 2621440000 [ 7.189381] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.196880] vde: detected capacity change from 0 to 4294967296 [ 7.227739] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.236023] vdf: detected capacity change from 0 to 4294967296 [ 7.251339] libphy: Fixed MDIO Bus: probed [ 7.264418] usbcore: registered new interface driver usbserial_generic [ 7.269290] usbserial: USB Serial support registered for generic [ 7.273416] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.290112] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.297346] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.307203] mousedev: PS/2 mouse device common for all mice [ 7.323978] rtc_cmos 00:05: RTC can wake from S4 [ 7.337718] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.340439] rtc_cmos 00:05: registered as rtc0 [ 7.352204] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.357385] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.370578] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.373738] intel_pstate: CPU model not supported [ 7.389904] hid: raw HID events driver (C) Jiri Kosina [ 7.392525] usbcore: registered new interface driver usbhid [ 7.396709] usbhid: USB HID core driver [ 7.399192] drop_monitor: Initializing network drop monitor service [ 7.403888] Initializing XFRM netlink socket [ 7.406275] NET: Registered protocol family 10 [ 7.409982] Segment Routing with IPv6 [ 7.412531] NET: Registered protocol family 17 [ 7.414765] mpls_gso: MPLS GSO support [ 7.422573] RAS: Correctable Errors collector initialized. [ 7.425533] AVX version of gcm_enc/dec engaged. [ 7.428039] AES CTR mode by8 optimization enabled [ 7.582897] sched_clock: Marking stable (7582865030, 0)->(8537365420, -954500390) [ 7.586799] registered taskstats version 1 [ 7.589878] Loading compiled-in X.509 certificates [ 7.595424] zswap: loaded using pool lzo/zbud [ 7.628693] Key type big_key registered [ 7.644766] Key type encrypted registered [ 7.646752] ima: No TPM chip found, activating TPM-bypass! [ 7.651434] ima: Allocated hash algorithm: sha1 [ 7.653615] ima: No architecture policies found [ 7.656242] evm: Initialising EVM extended attributes: [ 7.658770] evm: security.selinux [ 7.660391] evm: security.ima [ 7.661692] evm: security.capability [ 7.663190] evm: HMAC attrs: 0x1 [ 7.667991] rtc_cmos 00:05: setting system clock to 2026-08-22 07:27:50 UTC (1787383670) [ 7.676936] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 7.684238] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 7.692235] Write protecting the kernel read-only data: 28672k [ 7.697844] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 7.700975] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 7.776893] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.788645] systemd[1]: Detected virtualization kvm. [ 7.790921] systemd[1]: Detected architecture x86-64. [ 7.792929] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.838696] systemd[1]: No hostname configured. [ 7.842230] systemd[1]: Set hostname to . [ 7.848497] random: systemd: uninitialized urandom read (16 bytes read) [ 7.854814] systemd[1]: Initializing machine ID from random generator. [ 7.998669] random: ln: uninitialized urandom read (6 bytes read) [ 8.164870] random: systemd: uninitialized urandom read (16 bytes read) [ 8.168323] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.181753] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 8.185981] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.573777] device-mapper: uevent: version 1.0.3 [ 9.578958] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.117595] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 11.156053] random: fast init done [ 11.451565] scsi host0: ata_piix [ 11.492091] scsi host1: ata_piix [ 11.494782] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.498619] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.175351] random: crng init done [ 16.181141] random: 7 urandom warning(s) missed due to ratelimiting [ 16.691959] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.733758] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.612873] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.373928] SELinux: Disabled at runtime. [ 22.507359] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.538616] systemd[1]: Detected virtualization kvm. [ 22.543617] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.102600] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.108724] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.124621] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.131344] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.137231] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.156656] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.178680] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 24.545488] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 25.821866] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.750214] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.879983] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.238185] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.291533] EDAC sbridge: Ver: 1.1.2 [ 31.513584] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.238454] NFS: Registering the id_resolver key type [ 32.244168] Key type id_resolver registered [ 32.247792] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ 38.448014] hrtimer: interrupt took 5303062 ns [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg428-server login: [ 83.503386] libcfs: loading out-of-tree module taints kernel. [ 83.546178] Key type ._llcrypt registered [ 83.549147] Key type .llcrypt registered [ 83.703095] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 103.779890] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 105.392298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.399737] alg: No test for adler32 (adler32-zlib) [ 106.958979] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 108.028240] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 109.840483] Key type lgssc registered [ 112.257242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.669352] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 169.675727] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 182.943809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.983763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.296883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.354911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.503833] Lustre: lustre-MDT0000: new disk, initializing [ 184.610035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.644733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.845325] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.191167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.381763] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.427607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.432209] Lustre: Skipped 1 previous similar message [ 204.608516] Lustre: lustre-MDT0001: new disk, initializing [ 204.753585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.798612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.811859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.515237] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.222380] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 223.570349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.211360] Lustre: lustre-OST0000: new disk, initializing [ 224.221432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.232307] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.325722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.435604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.441967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.530642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.812188] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.125595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.309255] Lustre: lustre-OST0001: new disk, initializing [ 245.314526] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.320571] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.413639] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.956318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.966656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.008783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.704732] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.814724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.046310] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.651795] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.159369] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing yml_node [ 289.265585] Lustre: DEBUG MARKER: Client: 2.17.54.113 [ 292.771948] Lustre: DEBUG MARKER: MDS: 2.17.54.113 [ 295.658970] Lustre: DEBUG MARKER: OSS: 2.17.54.113 [ 297.545934] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:32:37 EDT 2026 [ 315.516039] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 317.238820] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.889091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.900201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.916351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.913104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.915047] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.438530] Lustre: server umount lustre-MDT0000 complete [ 344.190505] LustreError: 10331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384007 with bad export cookie 18213868566320996133 [ 344.199254] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 344.203108] LustreError: 10331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.151991] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.152240] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.152723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.152728] Lustre: Skipped 3 previous similar messages [ 349.175592] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 349.221853] Lustre: Skipped 3 previous similar messages [ 350.703912] Lustre: server umount lustre-MDT0001 complete [ 355.154584] Lustre: server umount lustre-OST0000 complete [ 359.061508] Lustre: server umount lustre-OST0001 complete [ 366.446801] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 376.996608] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 413.469377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.680640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.705101] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.797194] Lustre: lustre-MDT0000: new disk, initializing [ 413.886608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.899363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.069922] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 429.775934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.895722] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.956616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.959993] Lustre: Skipped 1 previous similar message [ 430.021858] Lustre: lustre-MDT0001: new disk, initializing [ 430.113125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.161370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.175612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.554550] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 444.507891] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 449.728442] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 456.266556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 456.651570] Lustre: lustre-OST0000: new disk, initializing [ 456.660989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 456.675180] Lustre: 17037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 456.795740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.420367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 458.431974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 458.555650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 463.190791] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 471.788839] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 476.430931] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.616436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.597280] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.869375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.321978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.329328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.343112] LustreError: Skipped 1 previous similar message [ 484.348131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.375331] Lustre: Skipped 1 previous similar message [ 484.389408] Lustre: Skipped 1 previous similar message [ 489.444128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.456776] Lustre: Skipped 2 previous similar messages [ 497.120051] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 497.402808] Lustre: server umount lustre-OST0000 complete [ 501.729143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 501.747366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.787229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.799646] Lustre: Skipped 2 previous similar messages [ 509.813841] Lustre: server umount lustre-MDT0000 complete [ 509.920354] LustreError: 15075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.931873] LustreError: 15075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 513.320068] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384176 with bad export cookie 18213868566320997666 [ 513.328836] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.330949] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 513.739621] Lustre: server umount lustre-MDT0001 complete [ 518.680992] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 03:36:19 (1787384179) [ 525.227772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.666603] LustreError: 19302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.758427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.300263] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 530.919368] LustreError: 19303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.035294] LustreError: 19302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.279209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.919786] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 549.939513] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 554.744965] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 560.659293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.539697] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 572.796248] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.417918] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 584.673890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 584.687460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.706646] Lustre: Skipped 1 previous similar message [ 584.710627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 584.717637] Lustre: Skipped 2 previous similar messages [ 589.192139] Lustre: server umount lustre-MDT0000 complete [ 590.817504] LustreError: 21310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.831386] LustreError: 21310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 592.710345] LustreError: 20039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384255 with bad export cookie 18213868566320998548 [ 592.713338] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.721046] LustreError: 20039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 593.093253] Lustre: server umount lustre-MDT0001 complete [ 607.139935] Lustre: server umount lustre-OST0000 complete [ 628.625520] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 638.978236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.334891] LustreError: 23683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.381270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.389642] Lustre: Skipped 2 previous similar messages [ 644.050142] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 651.817799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.818451] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 659.857807] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 666.694784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 673.062228] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 681.989346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.148685] Lustre: lustre-OST0001: new disk, initializing [ 682.151987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 682.157282] Lustre: 25997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 683.713126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 683.728950] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 683.816322] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 687.741686] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 698.625510] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 702.305646] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 715.745561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.747859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.757864] Lustre: Skipped 5 previous similar messages [ 715.765850] Lustre: Skipped 6 previous similar messages [ 721.397433] Lustre: server umount lustre-MDT0000 complete [ 724.737539] LustreError: 23664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384387 with bad export cookie 18213868566320999395 [ 724.739795] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.749317] LustreError: 23664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 725.077216] Lustre: server umount lustre-MDT0001 complete [ 738.957656] Lustre: server umount lustre-OST0000 complete [ 741.856106] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787384388/real 1787384388] req@ffff94b886671500 x1874207724256640/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787384404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 743.076595] Lustre: server umount lustre-OST0001 complete [ 758.925456] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 768.015448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.394573] LustreError: 29021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.422394] LustreError: 29021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 768.514662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.520750] Lustre: Skipped 3 previous similar messages [ 772.876204] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 781.464476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.615431] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 789.203042] Lustre: 30169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 795.089936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.827958] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 813.327916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.455528] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 827.068610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 830.791399] Lustre: 32048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 838.113863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 838.122458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.130216] Lustre: Skipped 1 previous similar message [ 838.137548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.140289] Lustre: Skipped 4 previous similar messages [ 843.362497] Lustre: server umount lustre-MDT0000 complete [ 845.795928] LustreError: 29022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.828143] LustreError: 29022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 846.464435] LustreError: 29002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384509 with bad export cookie 18213868566321000690 [ 846.477903] LustreError: 29002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 846.482282] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.718611] Lustre: server umount lustre-MDT0001 complete [ 861.231988] Lustre: server umount lustre-OST0000 complete [ 875.868146] Lustre: server umount lustre-OST0001 complete [ 887.220358] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 03:42:27 (1787384547) [ 918.322196] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 929.572664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.826916] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 943.857025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.745273] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 953.926606] Lustre: 36001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 961.584486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 968.668539] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 974.318490] LustreError: 36358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.341698] LustreError: 36358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 977.998238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 984.351119] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 992.655467] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 997.520792] Lustre: 37880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1004.023115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.043721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.055033] Lustre: Skipped 6 previous similar messages [ 1004.078109] Lustre: Skipped 7 previous similar messages [ 1008.537883] Lustre: server umount lustre-MDT0000 complete [ 1012.613944] LustreError: 34834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384675 with bad export cookie 18213868566321001796 [ 1012.626246] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.632261] LustreError: 34834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1013.124748] Lustre: server umount lustre-MDT0001 complete [ 1027.554483] Lustre: server umount lustre-OST0000 complete [ 1030.625075] Lustre: 3643:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787384677/real 1787384677] req@ffff94b9977a8a80 x1874207724356992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787384693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1031.530194] Lustre: server umount lustre-OST0001 complete [ 1040.975662] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 03:45:00 (1787384700) [ 1059.311843] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1070.244822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.728928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.741767] Lustre: Skipped 7 previous similar messages [ 1075.608102] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1085.311627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.067732] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1094.304480] Lustre: 41170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1101.194792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.647191] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1117.520251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.696775] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1131.142200] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1134.721661] Lustre: 43050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1135.555926] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1135.560420] Lustre: Skipped 1 previous similar message [ 1138.661303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.672557] Lustre: Skipped 2 previous similar messages [ 1144.493810] Lustre: server umount lustre-MDT0000 complete [ 1148.573716] LustreError: 40004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384811 with bad export cookie 18213868566321002895 [ 1148.580137] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.592261] LustreError: 40004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1155.159781] Lustre: server umount lustre-MDT0001 complete [ 1164.408678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.057528] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1184.195334] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1187.259989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.953148] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1192.968655] Lustre: Skipped 1 previous similar message [ 1193.038638] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1201.644528] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1203.433780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1209.723071] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1211.969766] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1218.543944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1224.655342] Lustre: server umount lustre-MDT0000 complete [ 1228.572256] LustreError: 40003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384891 with bad export cookie 18213868566321004001 [ 1228.582849] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.592771] LustreError: 40003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1229.166817] Lustre: server umount lustre-MDT0001 complete [ 1235.971839] Lustre: server umount lustre-OST0000 complete [ 1240.238901] Lustre: server umount lustre-OST0001 complete [ 1249.800936] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 03:48:29 (1787384909) [ 1270.355787] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1281.254780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.771949] LustreError: 47671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.815046] LustreError: 47671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 30 previous similar messages [ 1288.148644] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1299.597221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.228730] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1307.223753] Lustre: 48823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1313.839227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1320.461902] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1330.071656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.197889] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1345.214465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1350.729160] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1366.508377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.509358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1366.517857] Lustre: Skipped 27 previous similar messages [ 1368.440467] Lustre: server umount lustre-MDT0000 complete [ 1371.942258] LustreError: 48825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385034 with bad export cookie 18213868566321004820 [ 1371.964215] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.209953] Lustre: server umount lustre-MDT0001 complete [ 1387.141166] Lustre: server umount lustre-OST0000 complete [ 1400.486560] Lustre: server umount lustre-OST0001 complete [ 1410.855471] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 03:51:11 (1787385071) [ 1413.381829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1415.107753] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 03:51:15 (1787385075) [ 1426.602238] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 1436.482211] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1460.437658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.700684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1460.761739] Lustre: lustre-MDT0000: new disk, initializing [ 1460.845387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1465.384371] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1474.736536] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.470750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.834625] Lustre: lustre-OST0000: new disk, initializing [ 1482.839981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1482.845393] Lustre: Skipped 1 previous similar message [ 1482.852896] Lustre: 54900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1484.169047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1484.181912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1484.221570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1488.838869] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1498.593549] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1500.862779] Lustre: DEBUG MARKER: create test files [ 1535.463046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.463438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1535.478417] Lustre: Skipped 14 previous similar messages [ 1538.593665] Lustre: server umount lustre-OST0000 complete [ 1542.739655] Lustre: server umount lustre-MDT0000 complete [ 1554.189784] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1557.456765] Key type lgssc unregistered [ 1557.787790] LNet: 56528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.800584] LNetError: 56528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.818715] LNet: Removed LNI 192.168.204.128@tcp [ 1558.810279] Key type .llcrypt unregistered [ 1558.815939] Key type ._llcrypt unregistered [ 1588.589868] Key type ._llcrypt registered [ 1588.594828] Key type .llcrypt registered [ 1588.824326] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1589.956729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.994203] alg: No test for adler32 (adler32-zlib) [ 1591.214494] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1591.549843] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1593.304250] Key type lgssc registered [ 1594.714838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1602.782610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1602.816100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.345845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.664735] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1616.849853] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1623.290207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1623.636389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.826297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1629.909983] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1629.930725] Lustre: 59265:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1629.937235] Lustre: 59265:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1629.942710] Lustre: 59265:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1629.948070] Lustre: 59265:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1629.953939] Lustre: 59265:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1629.960943] Lustre: 59265:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1637.760846] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1644.187535] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1649.653289] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1686.497616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.497690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1686.510125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.650978] Lustre: server umount lustre-OST0000 complete [ 1690.823629] Lustre: server umount lustre-MDT0000 complete [ 1700.100246] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1702.408921] Key type lgssc unregistered [ 1702.652247] LNet: 60897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.659834] LNetError: 60897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.672057] LNet: Removed LNI 192.168.204.128@tcp [ 1703.355871] Key type .llcrypt unregistered [ 1703.357874] Key type ._llcrypt unregistered [ 1731.280388] Key type ._llcrypt registered [ 1731.284027] Key type .llcrypt registered [ 1731.438732] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1732.756665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.811809] alg: No test for adler32 (adler32-zlib) [ 1733.897069] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1734.291697] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1736.071181] Key type lgssc registered [ 1737.422147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1745.059229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1745.082347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.591676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.677439] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1757.529634] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.455247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1764.767124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.189917] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1771.954242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1772.035656] Lustre: 63712:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1772.043506] Lustre: 63712:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1772.049342] Lustre: 63712:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1772.054766] Lustre: 63712:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1772.069932] Lustre: 63712:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1772.083941] Lustre: 63712:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1778.526905] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1783.694327] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1792.489703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.498496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.507343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.600593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.788927] Lustre: server umount lustre-OST0000 complete [ 1801.628073] Lustre: server umount lustre-MDT0000 complete [ 1810.872644] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 1813.932890] Key type lgssc unregistered [ 1814.231610] LNet: 65270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.253599] LNetError: 65270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.332939] LNet: Removed LNI 192.168.204.128@tcp [ 1816.328494] Key type .llcrypt unregistered [ 1816.332149] Key type ._llcrypt unregistered [ 1832.332706] Key type ._llcrypt registered [ 1832.334988] Key type .llcrypt registered [ 1832.440219] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 1846.459787] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 1847.682474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.704665] alg: No test for adler32 (adler32-zlib) [ 1848.706212] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1848.891160] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 1850.519216] Key type lgssc registered [ 1851.635207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.146841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.172252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.354329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1884.382438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1884.448230] Lustre: lustre-MDT0000: new disk, initializing [ 1884.566488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.581538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1889.232631] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1899.436073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.603439] Lustre: 68421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1899.716768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1899.721972] Lustre: Skipped 1 previous similar message [ 1899.828778] Lustre: lustre-MDT0001: new disk, initializing [ 1899.981791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1900.010555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1900.031266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1905.566469] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1915.256789] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.379659] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1927.463080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.871669] Lustre: lustre-OST0000: new disk, initializing [ 1927.882772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1927.900406] Lustre: 70390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1928.059810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.916387] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 1937.484572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1937.495052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1937.627329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1943.404089] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1948.927957] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1949.190327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1953.956165] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1954.338327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1958.373766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.381675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.392907] LustreError: Skipped 1 previous similar message [ 1958.399069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.427963] Lustre: Skipped 1 previous similar message [ 1958.467736] Lustre: Skipped 1 previous similar message [ 1962.765938] Lustre: server umount lustre-OST0000 complete [ 1966.568187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1966.574712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.588897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.609171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.609820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.736249] Lustre: server umount lustre-MDT0000 complete [ 1975.632751] LustreError: 68414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385638 with bad export cookie 17693516546492731165 [ 1975.638630] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.643226] LustreError: 68414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1975.859782] Lustre: server umount lustre-MDT0001 complete [ 1983.462048] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 04:00:44 (1787385644) [ 1996.999459] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2038.958738] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2049.717749] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2084.726371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.953565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2084.976460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2085.047480] Lustre: lustre-MDT0000: new disk, initializing [ 2085.168799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.182511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2089.555726] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2099.238880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.356733] Lustre: 76422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2099.418967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2099.423894] Lustre: Skipped 1 previous similar message [ 2099.502898] Lustre: lustre-MDT0001: new disk, initializing [ 2099.638096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.672978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2099.683224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2103.551210] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2112.071073] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2116.643152] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2122.847125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2123.189828] Lustre: lustre-OST0000: new disk, initializing [ 2123.193721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2123.202426] Lustre: 78391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2123.310443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.818357] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2128.992816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2129.003766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2129.111377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2137.129616] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.393259] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2142.717727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2147.420953] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2147.656602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2149.861426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.863972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.879374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.885087] Lustre: Skipped 1 previous similar message [ 2149.888653] Lustre: Skipped 1 previous similar message [ 2155.020106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2155.026326] Lustre: Skipped 2 previous similar messages [ 2164.191159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2164.414306] Lustre: server umount lustre-OST0000 complete [ 2170.342939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.343919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.357251] Lustre: Skipped 1 previous similar message [ 2170.362290] Lustre: Skipped 2 previous similar messages [ 2174.233503] Lustre: server umount lustre-MDT0000 complete [ 2175.457996] LustreError: 77423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.478638] LustreError: 77423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2178.139993] LustreError: 76412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385840 with bad export cookie 17693516546492732047 [ 2178.156680] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.158379] LustreError: 76412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2178.604252] Lustre: server umount lustre-MDT0001 complete [ 2186.661797] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 04:04:07 (1787385847) [ 2192.410104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.778168] LustreError: 80803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.846102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.919299] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2197.987406] LustreError: 80804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.110333] LustreError: 80803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.171453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.424879] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2215.030428] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2220.435595] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2226.862048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2227.149682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.156800] Lustre: Skipped 1 previous similar message [ 2232.235282] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2238.198476] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.069910] Lustre: 83477:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 82444a17-dc92-45d1-91c3-167c7e088014 at adminstrative request [ 2242.536519] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.553595] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2252.773809] Lustre: MGS: Client 82444a17-dc92-45d1-91c3-167c7e088014 (at 0@lo) reconnecting [ 2252.782234] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xf58bff8e05321601 to 0xf58bff8e053218a1 [ 2252.791051] Lustre: MGC192.168.204.128@tcp: Connection restored to 0@lo (at 0@lo) [ 2277.415802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2277.656750] Lustre: lustre-OST0001: new disk, initializing [ 2277.666541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2277.677485] Lustre: 83869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2283.977867] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2286.158581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2286.170293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2286.274739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2293.011896] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2296.802612] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.805680] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.823501] LustreError: Skipped 1 previous similar message [ 2296.824846] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2296.824851] Lustre: Skipped 1 previous similar message [ 2296.867754] Lustre: Skipped 1 previous similar message [ 2310.111141] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2310.313677] Lustre: server umount lustre-OST0001 complete [ 2312.182486] LustreError: 83248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.204503] LustreError: 83248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2314.209529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2314.219392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.108679] Lustre: server umount lustre-OST0000 complete [ 2327.526942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2327.532698] LustreError: Skipped 1 previous similar message [ 2329.893832] Lustre: server umount lustre-MDT0000 complete [ 2332.644015] LustreError: 80800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.660701] LustreError: 80800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2333.094855] LustreError: 80784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385995 with bad export cookie 17693516546492733601 [ 2333.099783] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.109076] LustreError: 80784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2333.520853] Lustre: server umount lustre-MDT0001 complete [ 2341.461487] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 04:06:41 (1787386001) [ 2342.955666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2344.453850] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 04:06:45 (1787386005) [ 2345.959952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2347.649639] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 04:06:48 (1787386008) [ 2349.376659] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2351.154427] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 04:06:51 (1787386011) [ 2352.563422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2354.240810] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 04:06:54 (1787386014) [ 2355.679379] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2361.776220] loop: module loaded [ 2361.799675] loop0: detected capacity change from 0 to 4294967296 [ 2361.882417] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2363.399848] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2367.076927] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2370.435878] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2396.281050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.354341] LustreError: 87993:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2396.371353] LustreError: 87993:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2396.377333] LustreError: 87993:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2396.385637] LustreError: 87993:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2415.099257] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 2424.790669] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 2462.493077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.825486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2462.886883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2463.045445] Lustre: lustre-MDT0000: new disk, initializing [ 2463.156694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.163509] Lustre: Skipped 1 previous similar message [ 2463.188817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2468.010628] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2478.254310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.344588] Lustre: 90859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2478.552517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2482.851370] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2492.192984] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.203864] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2503.763253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.071075] Lustre: 92828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2510.467204] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2513.989979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2514.112084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2519.090098] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2523.274654] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2523.500991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2528.658752] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2528.780163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2534.882195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.901892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.928218] Lustre: Skipped 3 previous similar messages [ 2534.937118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.945038] Lustre: Skipped 12 previous similar messages [ 2536.806622] Lustre: server umount lustre-OST0000 complete [ 2556.383955] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2556.640378] Lustre: server umount lustre-MDT0000 complete [ 2559.463710] LustreError: 93551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.491184] LustreError: 93551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2560.404812] LustreError: 92827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386223 with bad export cookie 17693516546492734217 [ 2560.405282] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.423851] LustreError: 92827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2560.838808] Lustre: server umount lustre-MDT0001 complete [ 2569.960873] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 04:10:30 (1787386230) [ 2576.777764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.949789] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2590.783958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.318873] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2603.605524] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2609.347291] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2616.835271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.209637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.221647] Lustre: Skipped 4 previous similar messages [ 2624.649318] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2632.845435] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2637.659045] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2637.923932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2642.309658] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2642.558485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2648.664215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.837714] Lustre: lustre-OST0001: new disk, initializing [ 2648.839995] Lustre: Skipped 2 previous similar messages [ 2648.842747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2648.856222] Lustre: Skipped 3 previous similar messages [ 2648.867484] Lustre: 98612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2655.162437] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2655.798972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2655.813768] Lustre: Skipped 2 previous similar messages [ 2655.824342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2655.930390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2664.569847] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2669.169911] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2669.449973] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2674.153860] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2674.368313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2695.506516] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2721.378930] Lustre: Failing over lustre-MDT0000 [ 2722.788754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.793092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.804585] Lustre: Skipped 5 previous similar messages [ 2722.811176] Lustre: Skipped 9 previous similar messages [ 2723.866820] Lustre: server umount lustre-MDT0000 complete [ 2724.202370] LustreError: 95241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.229888] LustreError: 95241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2743.273591] Lustre: 66646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386390/real 1787386390] req@0000000030866388 x1874209549856384/t0(0) o400->MGC192.168.204.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787386406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2743.334319] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.329044] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2749.331024] LDISKFS-fs (dm-0): recovery complete [ 2749.346650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.529555] LustreError: 66643:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001441459b x1874209549863552/t0(0) o250->MGC192.168.204.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2755.081110] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2755.929057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2755.930520] Lustre: 95240:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2760.177213] Lustre: 95244:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2760.190091] Lustre: 95244:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message [ 2760.200807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2760.285902] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2760.529612] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2760.616144] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2760.956387] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2760.976233] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2761.296949] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2761.302433] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2761.309043] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2761.320754] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages [ 2761.963155] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2761.977691] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2762.324192] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2762.338154] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2763.321514] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2763.347098] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 5 previous similar messages [ 2764.011863] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2764.024381] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 5 previous similar messages [ 2764.353028] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2764.370677] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 5 previous similar messages [ 2767.640831] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2767.664782] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 12 previous similar messages [ 2768.306585] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2768.349291] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2768.679619] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2768.689835] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2775.748126] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2775.771615] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 24 previous similar messages [ 2776.432633] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2776.447546] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 24 previous similar messages [ 2776.767525] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2776.775047] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 24 previous similar messages [ 2791.991214] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2791.999654] Lustre: 101463:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 50 previous similar messages [ 2792.641710] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2792.661912] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 50 previous similar messages [ 2792.983555] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2792.996464] LustreError: 101463:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 50 previous similar messages [ 2815.271216] Lustre: 101463:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2815.282638] Lustre: 101463:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2815.292683] Lustre: 101463:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0efe3c3d-f864-49a3-bf66-93191393c53c@192.168.204.28@tcp [ 2815.309459] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2815.320509] Lustre: 101463:0:(ldlm_lib.c:1900:abort_req_replay_queue()) @@@ aborted: req@00000000467bc541 x1874209533922048/t0(8589934768) o35->0efe3c3d-f864-49a3-bf66-93191393c53c@192.168.204.28@tcp:243/0 lens 392/0 e 0 to 0 dl 1787386488 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2815.361091] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2815.385134] Lustre: lustre-MDT0000: Denying connection for new client 0efe3c3d-f864-49a3-bf66-93191393c53c (at 192.168.204.28@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2815.390139] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2815.446540] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2815.455260] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2815.457939] Lustre: Skipped 2 previous similar messages [ 2815.499119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2815.503773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2830.305103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2830.314280] LustreError: Skipped 2 previous similar messages [ 2835.150323] Lustre: server umount lustre-OST0000 complete [ 2836.966226] LustreError: 97067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.987293] LustreError: 97067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 2852.342653] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2852.350060] Lustre: Skipped 10 previous similar messages [ 2853.343157] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2853.594142] Lustre: server umount lustre-OST0001 complete [ 2861.539426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.557219] Lustre: Skipped 5 previous similar messages [ 2863.966201] Lustre: server umount lustre-MDT0000 complete [ 2868.385939] LustreError: 95224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386531 with bad export cookie 17693516546492785436 [ 2868.397204] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2868.407346] LustreError: 95224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2868.874959] Lustre: server umount lustre-MDT0001 complete [ 2879.153315] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 04:15:39 (1787386539) [ 2888.066220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.079872] Lustre: *** cfs_fail_loc=197, val=0*** [ 2903.711731] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2908.831764] LustreError: 103440:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 2913.986321] Lustre: *** cfs_fail_loc=197, val=0*** [ 2913.991546] Lustre: Skipped 65 previous similar messages [ 2934.560109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.571116] Lustre: Skipped 2 previous similar messages [ 2941.898430] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2950.093550] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2952.670275] Lustre: server umount lustre-OST0000 complete [ 2961.380603] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 04:17:01 (1787386621) [ 2982.556883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.205345] LustreError: 105063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.219642] LustreError: 105063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2988.135856] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 2998.615882] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3002.716788] LustreError: 105094:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3002.730197] LustreError: 105094:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3002.738062] LustreError: 105094:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3003.148810] Lustre: server umount lustre-MDT0000 complete [ 3036.433693] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3047.500993] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3087.715449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.067554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3088.088429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3088.163568] Lustre: lustre-MDT0000: new disk, initializing [ 3088.236844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3093.450397] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3103.954690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.033626] Lustre: 109274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3104.237232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3109.380489] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3119.402717] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.781553] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3132.992028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.439593] Lustre: 111243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3140.839946] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3142.351855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3151.322518] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3156.603957] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3156.883433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3162.229492] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3162.522830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3168.237654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.250457] Lustre: Skipped 1 previous similar message [ 3168.256757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3168.257488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.257492] Lustre: Skipped 3 previous similar messages [ 3168.286079] LustreError: Skipped 5 previous similar messages [ 3170.845924] Lustre: server umount lustre-OST0000 complete [ 3189.217925] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3189.535407] Lustre: server umount lustre-MDT0000 complete [ 3193.751709] LustreError: 109265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386856 with bad export cookie 17693516546492796006 [ 3193.754861] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.761768] LustreError: 109265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3194.062802] Lustre: server umount lustre-MDT0001 complete [ 3205.136930] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 04:21:04 (1787386864) [ 3214.847626] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3218.324511] Key type lgssc unregistered [ 3218.778639] LNet: 113731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.786599] LNetError: 113731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3218.826101] LNet: Removed LNI 192.168.204.128@tcp [ 3219.817853] Key type .llcrypt unregistered [ 3219.821820] Key type ._llcrypt unregistered [ 3240.994578] Key type ._llcrypt registered [ 3240.996989] Key type .llcrypt registered [ 3241.193701] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3259.202381] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3260.227391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3260.342731] alg: No test for adler32 (adler32-zlib) [ 3261.538535] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3261.905333] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3263.641281] Key type lgssc registered [ 3264.632677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.164490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.203275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.537530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3310.572065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3310.656852] Lustre: lustre-MDT0000: new disk, initializing [ 3310.722823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.738172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3315.616470] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3326.194099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3326.462514] Lustre: lustre-OST0000: new disk, initializing [ 3326.470755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3326.479572] Lustre: Skipped 1 previous similar message [ 3326.488302] Lustre: 117910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3326.491240] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3326.605756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.139212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3332.152368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3332.183838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3332.853504] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3343.583311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.881883] Lustre: lustre-OST0001: new disk, initializing [ 3343.889500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3343.896931] Lustre: 118770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3343.983616] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3350.950339] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3353.211035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3353.218887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3353.251934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3363.298785] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.302620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.312795] Lustre: Skipped 1 previous similar message [ 3363.327549] Lustre: Skipped 1 previous similar message [ 3367.405668] Lustre: server umount lustre-MDT0000 complete [ 3374.174113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3379.267884] LustreError: 117093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387042 with bad export cookie 11321205346502028298 [ 3379.272406] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3379.275000] LustreError: 117093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3379.433353] Lustre: server umount lustre-OST0000 complete [ 3383.454549] Lustre: server umount lustre-OST0001 complete [ 3392.225734] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 04:24:12 (1787387052) [ 3415.751489] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 04:24:36 (1787387076) [ 3425.784764] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3450.321816] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 04:25:10 (1787387110) [ 3493.121952] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 3503.041467] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3543.393568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.673171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3543.705936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3543.832926] Lustre: lustre-MDT0000: new disk, initializing [ 3543.958971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.993317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3548.692454] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3559.338127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.519528] Lustre: 126525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3559.588056] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3559.593309] Lustre: Skipped 1 previous similar message [ 3559.728253] Lustre: lustre-MDT0001: new disk, initializing [ 3559.959248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3560.060337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3560.084540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3564.631846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3576.937354] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3583.889893] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3591.416355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.797231] Lustre: lustre-OST0000: new disk, initializing [ 3591.803656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.810151] Lustre: 128494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3591.972196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.446404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3599.463693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3599.572243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3599.758634] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3610.031297] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3615.473323] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3615.664925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3620.566362] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3620.856094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3625.440765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3625.447151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.454271] LustreError: Skipped 1 previous similar message [ 3625.462403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3625.500351] Lustre: Skipped 1 previous similar message [ 3630.561380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3630.571141] Lustre: Skipped 2 previous similar messages [ 3635.685476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3635.703802] Lustre: Skipped 1 previous similar message [ 3637.215141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3637.444865] Lustre: server umount lustre-OST0000 complete [ 3641.826682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3641.837514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.921639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.928109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.934190] Lustre: Skipped 1 previous similar message [ 3647.806154] Lustre: server umount lustre-MDT0000 complete [ 3652.107723] LustreError: 127319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387314 with bad export cookie 11321205346502029523 [ 3652.112717] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.117120] LustreError: 127319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3652.447291] Lustre: server umount lustre-MDT0001 complete [ 3662.461353] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 04:28:42 (1787387322) [ 3669.159071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.501169] LustreError: 130905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.515353] LustreError: 130905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3669.576420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.977202] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3674.612059] LustreError: 130906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.717866] LustreError: 130905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.672187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.200826] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3697.560525] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3703.567913] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3711.057373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.568933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.584117] Lustre: Skipped 1 previous similar message [ 3719.353846] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3729.492533] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3735.439169] Lustre: *** cfs_fail_loc=159, val=0*** [ 3751.818521] Lustre: lustre-MDT0000: Client dbf50c2d-c6cc-40b2-b554-e4955057eff9 (at 192.168.204.28@tcp) reconnecting [ 3753.562738] Lustre: *** cfs_fail_loc=159, val=0*** [ 3753.566533] Lustre: Skipped 5 previous similar messages [ 3769.235724] Lustre: lustre-MDT0000: Client dbf50c2d-c6cc-40b2-b554-e4955057eff9 (at 192.168.204.28@tcp) reconnecting [ 3790.821244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3790.826240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.835835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3790.835843] Lustre: Skipped 1 previous similar message [ 3790.867103] Lustre: Skipped 1 previous similar message [ 3793.465735] Lustre: server umount lustre-OST0000 complete [ 3801.058226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.079468] Lustre: Skipped 1 previous similar message [ 3803.969308] Lustre: server umount lustre-MDT0000 complete [ 3806.177439] LustreError: 131634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.156214] LustreError: 130887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387470 with bad export cookie 11321205346502030398 [ 3808.168217] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3808.185320] LustreError: 130887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3808.711983] Lustre: server umount lustre-MDT0001 complete [ 3820.125925] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 3823.550511] Key type lgssc unregistered [ 3823.961608] LNet: 134685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3823.990811] LNetError: 134685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3824.007274] LNet: Removed LNI 192.168.204.128@tcp [ 3825.240564] Key type .llcrypt unregistered [ 3825.245555] Key type ._llcrypt unregistered [ 3856.461599] Key type ._llcrypt registered [ 3856.468276] Key type .llcrypt registered [ 3856.580496] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 3857.738978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3857.772057] alg: No test for adler32 (adler32-zlib) [ 3858.827847] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3858.994791] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 3860.623212] Key type lgssc registered [ 3862.175680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3870.334624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3870.354418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.766598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.371376] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3881.957557] LustreError: 136101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.001717] LustreError: 136101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3885.685276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.224172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3890.957516] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3898.604299] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3904.410722] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3911.484833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3911.928679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3917.040855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3918.587553] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 3929.540227] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3939.247852] Lustre: *** cfs_fail_loc=159, val=0*** [ 3955.623839] Lustre: lustre-MDT0000: Client 1f90d02a-43dd-45b9-bf12-cd7e8fa3a015 (at 192.168.204.28@tcp) reconnecting [ 3958.036707] Lustre: *** cfs_fail_loc=159, val=0*** [ 3973.510151] Lustre: lustre-MDT0001: Client 1f90d02a-43dd-45b9-bf12-cd7e8fa3a015 (at 192.168.204.28@tcp) reconnecting [ 3975.155856] Lustre: *** cfs_fail_loc=159, val=0*** [ 3975.159041] Lustre: Skipped 3 previous similar messages [ 3991.451238] Lustre: lustre-MDT0001: Client 1f90d02a-43dd-45b9-bf12-cd7e8fa3a015 (at 192.168.204.28@tcp) reconnecting [ 4007.887135] Lustre: 139162:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4011.224753] Lustre: *** cfs_fail_loc=159, val=0*** [ 4011.234534] Lustre: Skipped 4 previous similar messages [ 4027.825587] Lustre: lustre-MDT0000: Client df68d680-7f6d-47f6-a5d3-a2a96262bfd8 (at 192.168.204.28@tcp) reconnecting [ 4030.212080] Lustre: *** cfs_fail_loc=159, val=0*** [ 4030.214016] Lustre: Skipped 14 previous similar messages [ 4046.796874] Lustre: lustre-MDT0000: Client df68d680-7f6d-47f6-a5d3-a2a96262bfd8 (at 192.168.204.28@tcp) reconnecting [ 4048.635292] Lustre: 139452:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4055.041138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4055.047137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.047977] LustreError: Skipped 2 previous similar messages [ 4055.083879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4058.597573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4058.609026] Lustre: Skipped 2 previous similar messages [ 4060.120038] Lustre: server umount lustre-OST0000 complete [ 4065.248827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4065.256333] LustreError: Skipped 4 previous similar messages [ 4065.263106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.293284] Lustre: Skipped 1 previous similar message [ 4065.300233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4068.833672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4068.837166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.856203] Lustre: server umount lustre-MDT0000 complete [ 4073.957817] LustreError: 136395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.988329] LustreError: 136395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4076.092588] LustreError: 139037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387738 with bad export cookie 8880736278415879945 [ 4076.109624] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.114038] LustreError: 139037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4077.056157] Lustre: server umount lustre-MDT0001 complete [ 4091.847776] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4095.378828] Key type lgssc unregistered [ 4095.806631] LNet: 140507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4095.824731] LNetError: 140507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4095.846199] LNet: Removed LNI 192.168.204.128@tcp [ 4096.633321] Key type .llcrypt unregistered [ 4096.637801] Key type ._llcrypt unregistered [ 4120.734034] Key type ._llcrypt registered [ 4120.740171] Key type .llcrypt registered [ 4120.858510] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4121.692863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4121.892926] alg: No test for adler32 (adler32-zlib) [ 4122.914896] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4123.202712] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4124.975739] Key type lgssc registered [ 4126.487768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4135.387357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4135.406247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.014526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.131363] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4147.189126] LustreError: 141927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.228659] LustreError: 141927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4151.486626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.990745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4156.926087] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4164.478731] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4171.222674] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4179.454142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.900642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.989533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4188.055033] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4196.312319] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4196.523095] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4203.330731] Lustre: 144513:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4207.137249] Lustre: 144561:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4209.039963] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4210.164068] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4214.425793] Lustre: 144707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4215.511406] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4216.582471] Lustre: 144806:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4221.920597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.926563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.940392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.977713] Lustre: Skipped 1 previous similar message [ 4227.052492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4227.073948] Lustre: Skipped 2 previous similar messages [ 4227.702532] Lustre: server umount lustre-OST0000 complete [ 4232.165121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.188704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4232.199200] Lustre: Skipped 1 previous similar message [ 4237.283817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.287263] Lustre: Skipped 1 previous similar message [ 4240.083616] Lustre: server umount lustre-MDT0000 complete [ 4242.399896] LustreError: 142655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.411919] LustreError: 142655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4244.374745] LustreError: 141908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387907 with bad export cookie 7906790186135381878 [ 4244.388551] LustreError: 141908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4244.391837] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4244.769275] Lustre: server umount lustre-MDT0001 complete [ 4255.986628] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4258.727900] Key type lgssc unregistered [ 4259.067100] LNet: 145867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4259.073806] LNetError: 145867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4260.139377] LNet: Removed LNI 192.168.204.128@tcp [ 4261.026538] Key type .llcrypt unregistered [ 4261.029901] Key type ._llcrypt unregistered [ 4285.697185] Key type ._llcrypt registered [ 4285.698810] Key type .llcrypt registered [ 4285.833634] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4287.242806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4287.283615] alg: No test for adler32 (adler32-zlib) [ 4288.409883] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4288.644762] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4290.392512] Key type lgssc registered [ 4291.626587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4299.874249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4299.886874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.361817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.556540] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4311.527427] LustreError: 147281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.542622] LustreError: 147281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4313.687613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.103857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4318.517861] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4326.284984] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4332.235687] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4339.743821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4340.282513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.426855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4346.944871] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4354.304682] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4358.717224] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4363.929593] Lustre: *** cfs_fail_loc=159, val=0*** [ 4379.528584] Lustre: lustre-MDT0000: Client 3b2e54bf-b31e-43e4-8433-86eb67e394ad (at 192.168.204.28@tcp) reconnecting [ 4397.539723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4397.551826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.569205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4399.586600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4399.593154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4399.594882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.602859] Lustre: Skipped 1 previous similar message [ 4403.374403] Lustre: server umount lustre-OST0000 complete [ 4409.832463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.852380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4409.860262] Lustre: Skipped 1 previous similar message [ 4413.313431] Lustre: server umount lustre-MDT0000 complete [ 4414.945300] LustreError: 147282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.968871] LustreError: 147282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4417.073750] LustreError: 148022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388079 with bad export cookie 3057678522472613881 [ 4417.085975] LustreError: 148022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4417.089511] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.441477] Lustre: server umount lustre-MDT0001 complete [ 4428.250484] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4430.890966] Key type lgssc unregistered [ 4431.255277] LNet: 150949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4431.268135] LNetError: 150949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4431.296353] LNet: Removed LNI 192.168.204.128@tcp [ 4432.238916] Key type .llcrypt unregistered [ 4432.240819] Key type ._llcrypt unregistered [ 4456.834388] Key type ._llcrypt registered [ 4456.837425] Key type .llcrypt registered [ 4456.923381] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4457.947055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4458.106505] alg: No test for adler32 (adler32-zlib) [ 4459.170618] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4459.479953] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4461.184910] Key type lgssc registered [ 4462.410512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4469.980101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4470.004141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.611377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.552840] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4482.020376] LustreError: 152363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.036977] LustreError: 152363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4485.196185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.579293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4490.254566] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4498.455846] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4504.023215] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4510.378658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4510.721244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.856581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4516.979105] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4523.998562] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4531.228680] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4550.113318] Lustre: lustre-MDT0001: evicting 70798867-fa90-48f7-866b-57340b4bb8da (at 192.168.204.28@tcp) by administrative request [ 4550.120811] Lustre: Skipped 1 previous similar message [ 4582.368726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.377942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.384341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.408874] Lustre: Skipped 1 previous similar message [ 4584.843788] Lustre: server umount lustre-OST0000 complete [ 4592.608672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.617723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.629222] Lustre: Skipped 1 previous similar message [ 4592.631377] Lustre: Skipped 1 previous similar message [ 4594.705827] Lustre: server umount lustre-MDT0000 complete [ 4597.729586] LustreError: 152359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.744879] LustreError: 152359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4598.246218] LustreError: 152343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388261 with bad export cookie 8619909425313781194 [ 4598.250573] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.258600] LustreError: 152343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4598.563730] Lustre: server umount lustre-MDT0001 complete [ 4609.213257] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4611.748956] Key type lgssc unregistered [ 4612.052217] LNet: 156436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4612.069399] LNetError: 156436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4612.095740] LNet: Removed LNI 192.168.204.128@tcp [ 4612.924206] Key type .llcrypt unregistered [ 4612.926146] Key type ._llcrypt unregistered [ 4681.024655] Key type ._llcrypt registered [ 4681.029530] Key type .llcrypt registered [ 4681.169841] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4682.437640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4682.491591] alg: No test for adler32 (adler32-zlib) [ 4683.624418] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4683.901978] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4685.607227] Key type lgssc registered [ 4687.194710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4695.775490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4695.791722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.343346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4701.817651] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4707.815919] LustreError: 158925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.847204] LustreError: 158925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4710.854947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.331546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4716.530780] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4723.826804] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4729.289134] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.967289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4736.419195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.466355] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4742.577572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4743.209507] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4751.735203] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4763.110437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4763.120286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.120426] LustreError: Skipped 1 previous similar message [ 4763.146587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4768.230111] LustreError: 160753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.324861] Lustre: server umount lustre-OST0000 complete [ 4772.840518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4772.861420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.873191] Lustre: Skipped 1 previous similar message [ 4772.883586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.893761] Lustre: Skipped 1 previous similar message [ 4778.470670] LustreError: 159655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.495742] LustreError: 159655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4780.491134] Lustre: server umount lustre-MDT0000 complete [ 4783.588025] LustreError: 158924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.605030] LustreError: 158924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4784.290301] LustreError: 158906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388447 with bad export cookie 10417226291189778093 [ 4784.298484] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4784.304216] LustreError: 158906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4784.726578] Lustre: server umount lustre-MDT0001 complete [ 4795.244824] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 4798.233789] Key type lgssc unregistered [ 4798.504966] LNet: 162467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4798.512862] LNetError: 162467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4798.529376] LNet: Removed LNI 192.168.204.128@tcp [ 4799.404220] Key type .llcrypt unregistered [ 4799.411784] Key type ._llcrypt unregistered [ 4823.464488] Key type ._llcrypt registered [ 4823.467427] Key type .llcrypt registered [ 4823.644100] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4824.783621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4824.824562] alg: No test for adler32 (adler32-zlib) [ 4826.190879] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4826.558193] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 4828.335183] Key type lgssc registered [ 4829.486754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4844.137254] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 04:48:24 (1787388504) [ 4860.533149] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 4872.068470] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 4909.944609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4909.969673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.157311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4911.181462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4911.257216] Lustre: lustre-MDT0000: new disk, initializing [ 4911.334375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.375564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4916.420313] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4926.437089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.575177] Lustre: 166737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4926.683763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4926.691504] Lustre: Skipped 1 previous similar message [ 4926.768803] Lustre: lustre-MDT0001: new disk, initializing [ 4926.887673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4926.937574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4926.945772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4931.153573] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4940.541604] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4946.624201] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4952.838707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4953.113777] Lustre: lustre-OST0000: new disk, initializing [ 4953.116739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4953.121537] Lustre: 168706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4953.190696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4959.862406] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 4962.420556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4962.429841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4962.527138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4969.349913] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4983.268161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.269497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4983.296440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4983.306345] Lustre: Skipped 1 previous similar message [ 4985.118699] Lustre: server umount lustre-OST0000 complete [ 4993.519641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.520456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4993.534658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4993.535571] Lustre: Skipped 2 previous similar messages [ 4995.150511] Lustre: server umount lustre-MDT0000 complete [ 4999.648208] LustreError: 166729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388662 with bad export cookie 8502776379350312354 [ 4999.655822] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4999.676056] LustreError: 166729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5000.015575] Lustre: server umount lustre-MDT0001 complete [ 5012.476536] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 5015.848320] Key type lgssc unregistered [ 5016.087451] LNet: 170612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5016.098706] LNetError: 170612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5016.119411] LNet: Removed LNI 192.168.204.128@tcp [ 5017.005200] Key type .llcrypt unregistered [ 5017.008479] Key type ._llcrypt unregistered [ 5042.610863] Key type ._llcrypt registered [ 5042.613140] Key type .llcrypt registered [ 5042.733537] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 5043.820436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5043.940082] alg: No test for adler32 (adler32-zlib) [ 5045.009437] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 5045.281533] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 5047.032901] Key type lgssc registered [ 5048.258635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5056.476311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5056.502865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.117817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.259422] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5067.237518] LustreError: 172027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.254080] LustreError: 172027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5071.608861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.933245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5075.868944] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5082.330435] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5087.313540] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5092.667185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5092.980423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.368977] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 5102.268500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5104.481121] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5111.075276] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5111.094842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5111.112574] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5111.121317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.132654] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5112.049446] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5112.072402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.089691] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5112.098324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.128870] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5113.130456] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5113.144113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.155897] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5113.161760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.169985] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5114.006939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.951496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.684276] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5115.694607] Lustre: Skipped 2 previous similar messages [ 5115.710507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.717304] Lustre: Skipped 2 previous similar messages [ 5115.720141] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5115.723906] Lustre: Skipped 2 previous similar messages [ 5115.727761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.737776] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5115.745295] Lustre: Skipped 2 previous similar messages [ 5116.596236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.464837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.362493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.305538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.357785] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5120.360919] Lustre: Skipped 4 previous similar messages [ 5120.380064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.395324] Lustre: Skipped 4 previous similar messages [ 5120.399728] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5120.405596] Lustre: Skipped 4 previous similar messages [ 5120.410752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.425580] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5120.434053] Lustre: Skipped 4 previous similar messages [ 5121.275654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.246237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.133791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.028923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.099688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.892288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.878079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.733952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.675975] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5128.682491] Lustre: Skipped 8 previous similar messages [ 5128.696984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.704637] Lustre: Skipped 8 previous similar messages [ 5128.707929] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5128.712262] Lustre: Skipped 8 previous similar messages [ 5128.715640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.723103] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5128.728602] Lustre: Skipped 8 previous similar messages [ 5129.757309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.719080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.539298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.276121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.219561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.198942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.975168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.039476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.960703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.081943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.102077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.891780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.737657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.427351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.309250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.313095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.386940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.323521] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5145.328787] Lustre: Skipped 17 previous similar messages [ 5145.350880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.368073] Lustre: Skipped 17 previous similar messages [ 5145.377063] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5145.380977] Lustre: Skipped 17 previous similar messages [ 5145.384572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.394985] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5145.401327] Lustre: Skipped 17 previous similar messages [ 5146.393492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.298555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.355482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.347779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.148335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.036992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.077709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.031434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.058871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.139187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.133395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.343656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.479308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.346276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.396863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.454275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.533448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.552600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.457912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.440368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.425210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.359782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.356097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.220540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.121954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.092823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.900825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.678958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.404089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.217424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.960800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.807887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.721856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.580655] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5177.588459] Lustre: Skipped 33 previous similar messages [ 5177.599962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.614978] Lustre: Skipped 33 previous similar messages [ 5177.618870] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5177.622508] Lustre: Skipped 33 previous similar messages [ 5177.625660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.633143] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5177.638757] Lustre: Skipped 33 previous similar messages [ 5178.428098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.351919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.262715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.047803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.891161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.819055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.642846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.499462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.295825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.168058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.898830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.786867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.475135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.456785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.363616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.338629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.139661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.190550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.331573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.183071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.194387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.067464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.147880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.254786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.163162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.224673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.021482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.015540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.999387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.096726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.933329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.966631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.840680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.840165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.784080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.942046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.921449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.139782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.209365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.235984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.221549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.145162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.927121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.872539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.057366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.915258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.821495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.765696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.793113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.802919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.688823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.479151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.342321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.200911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.072624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.997679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.946409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.729223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.702415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.447058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.456588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.393481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.403400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.202262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.193483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.087382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.977380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.885370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.803476] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5241.811257] Lustre: Skipped 68 previous similar messages [ 5241.831594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.845969] Lustre: Skipped 68 previous similar messages [ 5241.852337] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5241.858095] Lustre: Skipped 68 previous similar messages [ 5241.867281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.875772] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5241.883182] Lustre: Skipped 68 previous similar messages [ 5242.751915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.473629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.368476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.269353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.110494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.902122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.758074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.807541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.661986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.685362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.494382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.427645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.540767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.557334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.484232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.355805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.250707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.279838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.092385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.974812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.798963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.662389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.459213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.253391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.073453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.060285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.858791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.660249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.531782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.267600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.144782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.050880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.070294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.862420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.752811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.700182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.512860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.359198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.101678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.857718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.828402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.951538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.647235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.473371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.634145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.528200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.807779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.966070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.956888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.213618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.250950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.343771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.362451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.247287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.271845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.155860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.338388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.502220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.591278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.520813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.750240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.234837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.451620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.397354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.119979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.996461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.020253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.007771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.350440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.439903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.947200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.642234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.867597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.921731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.201115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.993146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.917604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.680418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.800855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.943855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.179419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.773493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.912344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.056175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.228854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.261856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.397068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.540633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.450225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.414407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.608989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.358748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.281171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.410709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.516973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.453732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.720534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.017546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.216811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.269773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.186712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.372832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.552430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.536376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.629710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.675586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.963862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.239468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.347590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.609355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.702472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.700796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.726900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.825397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.985885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.981448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.173813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.058172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.086359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.152704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.975267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.940296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.908315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.846475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.725402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.609675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.329858] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5370.349571] Lustre: Skipped 126 previous similar messages [ 5370.393366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.414300] Lustre: Skipped 126 previous similar messages [ 5370.418466] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5370.425784] Lustre: Skipped 126 previous similar messages [ 5370.429640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.441735] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5370.461178] Lustre: Skipped 126 previous similar messages [ 5371.563745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.493775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.529183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.660926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.456817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.365829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.200236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.279641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.114085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.049332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.874637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.060758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.980493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.736926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.621336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.516439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.570492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.523494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.356576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.324154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.258714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.304396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.424369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.427453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.356478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.188906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.108332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.218604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.255830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.456186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.323427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.316563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.261220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.150917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.009169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.293917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.336158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.170068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.109982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.108712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.022988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.034091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.958619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.752368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.544598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.387521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.280575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.224739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.237105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.078264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.100928] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5419.108246] LustreError: 172047:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5419.113298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.107686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.036909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.861135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.656182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.451339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.340093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.310170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.196340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.071540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.022625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.866385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.778437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.740332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.645258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.385533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.252900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.028451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.924081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.801428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.713187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.801275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.670314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.416749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.366412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.308888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.232490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.890331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.752411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.678671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.545654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.675264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.563803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.466565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.411541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.344872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.256688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.199152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.037470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.902142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.587503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.501975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.319469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.168773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.005811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.022964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.867055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.833450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.712594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.661146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.392851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.260648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.137946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.010439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.881396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.728069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.518488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.373784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.215325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.092650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.906384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.929265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.856260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.991553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.986530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.891392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.758278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.672564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.614322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.517663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.306324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.175189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.060733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.964313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.791878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.753893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.701602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.606645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.487899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.534104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.407587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.456268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.389315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.541781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.448896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.505282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.462186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.607584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.463903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.160711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.029275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.268533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.260486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.291202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.207854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.062984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.020470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.914825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.039994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.060100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.051327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.173472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.473771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.857734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.034072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.039922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.224203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.430560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.605254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.885311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.696593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.601767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.610535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.607748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.576645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.574578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.519581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.361572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.359617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.377387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.365148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.392542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.437544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.338327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.316326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.260460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.362160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.173887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.193209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.235699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.128749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.032849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.076919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.060873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.998810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.866353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.681306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.525977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.343916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.170875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.142218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.149267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.192675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.381076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.245200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.213386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.245361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.322056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.408944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.321390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.384315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.461784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.419965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.261126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.200626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.289630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.454363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.394692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.413410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.391901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.368469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.276538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.430519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.379338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.347668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.289044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.360868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.326599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.188046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.112497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.072177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.012917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.007405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.886384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.948205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.909060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.846474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.588560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.304138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.094083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.948350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.830674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.699921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.612071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.505874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.489393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.412539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.217326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.055922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.949691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.892714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.764651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.774952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.722646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.506469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.437832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.259358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.233652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.110900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.967625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.916772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.826786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.726331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.647706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.350219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.244056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.084531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.102622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.061177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.060544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.828485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.646870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.567884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.283793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.105310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.054322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.974445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.893417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.850886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.111873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.106704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.099878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.317816] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5627.325747] Lustre: Skipped 272 previous similar messages [ 5627.343587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.356804] Lustre: Skipped 272 previous similar messages [ 5627.363546] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5627.371386] Lustre: Skipped 272 previous similar messages [ 5627.376674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.387784] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5627.393542] Lustre: Skipped 272 previous similar messages [ 5628.310697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.354491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.512730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.491221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.429277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.484548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.449273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.486166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.305899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.206627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.155251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.972919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.974348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.998454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.108948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.181059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.075885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.079549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.992584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.795653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.794304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.640315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.525416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.329608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.093506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.763070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.504852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.273072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.092283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.920536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.919694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.721709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.603801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.349695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.291742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.312327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.246263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.081178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.851934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.727846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.491864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.249385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.053423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.874403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.691444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.587736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.606417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.587547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.503552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.451882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.208520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.045396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.858429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.823253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.675332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.571771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.591817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.693959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.558452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.665331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.554858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.613522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.632849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.623230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.469665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.331281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.289567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.299328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.258690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.208889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.049261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.087870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.920858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.887362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.852220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.984833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.937144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.814683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.882658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.767190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.696503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.558723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.475617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.523729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.443610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.301686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.172504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.897853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.799138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.721729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.529073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.324472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.252123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.283754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.086546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.980763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.902407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.771155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.700139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.636667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.557612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.582983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.518092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.429392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.458048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.226932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.025374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.830797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.596502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.581728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.463166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.198120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.088522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.983907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.874235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.852549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.782988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.583548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.455577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.328447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.332912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.220560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.081231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.939585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.000663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.937525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.836405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.693053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.544515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.364357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.343466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.096627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.955134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.798347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.550867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.463041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.245171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.057675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.908687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.739919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.595979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.453266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.114454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.812763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.645038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.493848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.285598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.130975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.959853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.814234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.742916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.779228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.622800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.515493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.314733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.090590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.886833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.772690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.474415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.333537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.245753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.184228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.983858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.690058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.362320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.980756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.779173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.554544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.373642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.233426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.185198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.017093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.891251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.807152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.740560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.615304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.644682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.521943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.352415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.254249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.025222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.879565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.786147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.610928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.615620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.501710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.490709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.373263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.311501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.163036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.044029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.133761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.020067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.902270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.842598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.783496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.707370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.586778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.508917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.442275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.190028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.995191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.695831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.557394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.472643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.257316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.047268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.935475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.935272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.996603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.815497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.918390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.666731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.501326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.458693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.532772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.401581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.389405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.433974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.459954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.600719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.526321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.735857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.025694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.139850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.065297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.153231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.117331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.636221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.919376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.185526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.278779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.284548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.217064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.508568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.626294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.555147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.500217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.537715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.596878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.445574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.730737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.700533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.675115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.650792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.692030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.487982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.388088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.341215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.198162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.921968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.842971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.729773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.719032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.731475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.576353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.424724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.321560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.212767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.161931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.073531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.950885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.731957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.554265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.385863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.283558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.256580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.087741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.940610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.806479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.705396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.631329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.472143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.377228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.298577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.244218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.159947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.054564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.785297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.836244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.906613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.967694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.903264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.899329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.916928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.843134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.782707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.789132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.745047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.476714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.473855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.323967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.167634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.036186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.808512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.642895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.564265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.570634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.424307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.245481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.323447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.239455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.137909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.912558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.945751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.852380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.912183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.738958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.693205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.631592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.507942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.306413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.240199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.158522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.132762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.055514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.017572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.845541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.675627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.389409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.214526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.053073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.878793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.685110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.524927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.361259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.456528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.454937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.525840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.549385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.535191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.354095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.255443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.058751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.005859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.926043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.918878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.826563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.937885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.767734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.717987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.572562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.542554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.534950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.413683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.275602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.224376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.263525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.161146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.082349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.107184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.060204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.850628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.946393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.772570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.820167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.748879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.633317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.554464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.532512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.418241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.450844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.284437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.374622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.237662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.208281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.151820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.229313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.156734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.053600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.962180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.943785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.921169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.800968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.655279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.462627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.382976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.391985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.276085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.227954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.026125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.136019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.036318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.936713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.875435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.772787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.646675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.489838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.299356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.182746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.123246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.071542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.998686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.794489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.680358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.494987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.270205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.063601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.985431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.835473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.806379] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5992.826365] LustreError: 172047:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5992.839227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.718392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.636843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.465314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.484679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.417088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.435526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.331703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.395360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.371640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.225092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.180334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.053367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.907875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.835641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.699629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.680561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.467714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.409783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.328894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.417176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.207436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.145517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.167643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.113266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.966704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.894870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.733544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.556947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.381391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.364705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.468845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.369226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.342967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.112497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.915542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.745857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.834034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.049988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.078591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.066240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.095119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.979550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.090942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.116191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.927757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.032406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.818131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.787968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.766895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.624496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.417170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.284598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.206942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.112854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.051780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.938893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.687817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.923179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.259447] LustreError: 173851:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6391.265626] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7051.235900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7051.251584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.265842] Lustre: Skipped 459 previous similar messages [ 7051.271311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7053.281266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7053.298941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7054.312377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7054.317215] Lustre: Skipped 1 previous similar message [ 7055.775546] Lustre: server umount lustre-OST0000 complete [ 7063.525700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7063.535267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7069.668730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7069.679387] Lustre: Skipped 2 previous similar messages [ 7074.272785] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7074.685865] Lustre: server umount lustre-MDT0000 complete [ 7074.796262] LustreError: 172024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.818391] LustreError: 172024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7078.879040] LustreError: 172007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390741 with bad export cookie 4856449077689327096 [ 7078.885181] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7078.894938] LustreError: 172007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7079.274553] Lustre: server umount lustre-MDT0001 complete [ 7091.607501] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7094.554325] Key type lgssc unregistered [ 7094.925397] LNet: 227281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7094.939611] LNetError: 227281:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7095.974605] LNet: Removed LNI 192.168.204.128@tcp [ 7097.091317] Key type .llcrypt unregistered [ 7097.098530] Key type ._llcrypt unregistered [ 7128.258346] Key type ._llcrypt registered [ 7128.263967] Key type .llcrypt registered [ 7128.420980] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7129.662681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7129.852136] alg: No test for adler32 (adler32-zlib) [ 7131.081297] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7131.273731] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7132.943623] Key type lgssc registered [ 7134.182771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7142.006492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7142.036863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.713761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7148.371547] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7154.147036] LustreError: 228713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.184590] LustreError: 228713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7156.221864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.500410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7161.131080] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7169.032464] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7174.469891] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7181.582877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7181.974097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7188.067057] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7189.177902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7197.471552] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7206.571538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7208.722095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.761944] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7208.776463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7208.800464] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7208.801822] LustreError: 228733:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7212.006564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7212.020054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.040201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7214.386158] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 7215.083489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.544802] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 7219.550752] Lustre: Skipped 2 previous similar messages [ 7224.666402] Lustre: lustre-OST0000: Not available for connect from 192.168.204.28@tcp (stopping) [ 7224.677906] Lustre: Skipped 2 previous similar messages [ 7225.312792] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7225.602317] Lustre: server umount lustre-OST0000 complete [ 7228.810695] LustreError: 228733:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7260.291798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7260.813582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.746637] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7276.653472] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7280.324384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7280.354239] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7280.358565] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7280.385764] Lustre: Skipped 1 previous similar message [ 7281.888931] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7286.840632] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fd59949e800.ost_server_uuid 50 [ 7288.275812] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9fd59949e800.ost_server_uuid in FULL state after 0 sec [ 7294.498696] Lustre: server umount lustre-OST0000 complete [ 7300.064769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7300.077298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.099204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.106868] Lustre: Skipped 2 previous similar messages [ 7308.351477] Lustre: server umount lustre-MDT0000 complete [ 7311.330175] Lustre: 228027:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390958/real 1787390958] req@0000000041d2eaba x1874215089278848/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787390974 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7311.345897] LustreError: 228708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.373769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.385115] LustreError: 228708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7311.413147] Lustre: Skipped 1 previous similar message [ 7312.665111] LustreError: 230556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390975 with bad export cookie 4493939324807758096 [ 7312.667125] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.682322] LustreError: 230556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7313.158118] Lustre: server umount lustre-MDT0001 complete [ 7323.628366] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 7326.713385] Key type lgssc unregistered [ 7327.128195] LNet: 233924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7327.141091] LNetError: 233924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7327.181035] LNet: Removed LNI 192.168.204.128@tcp [ 7328.271197] Key type .llcrypt unregistered [ 7328.273411] Key type ._llcrypt unregistered [ 7364.507770] Key type ._llcrypt registered [ 7364.510021] Key type .llcrypt registered [ 7364.612743] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7366.133661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7366.221978] alg: No test for adler32 (adler32-zlib) [ 7367.343833] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7367.656081] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 7369.344828] Key type lgssc registered [ 7370.702741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7379.682069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7379.694320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.070757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.027572] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7390.177615] LustreError: 235656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.192751] LustreError: 235656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7393.818723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.163852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7398.994273] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7402.053485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7409.526655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7409.944055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.652374] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7425.339397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7426.599284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7426.800851] Lustre: lustre-OST0001: new disk, initializing [ 7426.806176] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7426.812958] Lustre: 237964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7426.927921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7434.173618] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7436.426456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7436.443169] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7436.578474] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7444.986986] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7451.238938] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7465.953036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7465.963061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7465.986987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7467.493880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7467.494913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7467.524257] Lustre: Skipped 2 previous similar messages [ 7471.777322] Lustre: server umount lustre-MDT0000 complete [ 7472.613302] LustreError: 236383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7472.642621] LustreError: 236383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7475.153686] LustreError: 236790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391137 with bad export cookie 5058867184736234794 [ 7475.160136] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7475.163244] LustreError: 236790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7475.493210] Lustre: server umount lustre-MDT0001 complete [ 7489.269169] Lustre: server umount lustre-OST0000 complete [ 7502.486271] Lustre: server umount lustre-OST0001 complete [ 7524.745658] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7537.030484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.647198] LustreError: 241069:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.725521] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.229346] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7542.760935] LustreError: 241070:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.885288] LustreError: 241069:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.881930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.269955] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7556.334372] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7559.193246] Lustre: 242214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7565.142679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.674855] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7575.744027] LustreError: 242569:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.721646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7579.891239] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7579.894608] Lustre: Skipped 1 previous similar message [ 7582.970745] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7582.977110] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7585.729401] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7592.967319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7597.122867] Lustre: 244083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7607.776367] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7607.784700] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.803908] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.809895] Lustre: Skipped 3 previous similar messages [ 7613.929251] LustreError: 241070:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.957246] LustreError: 241070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7614.001145] Lustre: server umount mylustre-MDT0000 complete [ 7617.487348] LustreError: 241046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391280 with bad export cookie 5058867184736237601 [ 7617.490382] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7617.504846] LustreError: 241046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7617.920979] Lustre: server umount mylustre-MDT0001 complete [ 7631.309123] Lustre: server umount mylustre-OST0000 complete [ 7634.400498] Lustre: 234916:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391281/real 1787391281] req@ffff94b8a79f6a00 x1874215336823296/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787391297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7634.452943] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.485183] Lustre: Skipped 3 previous similar messages [ 7634.889779] Lustre: server umount mylustre-OST0001 complete [ 7657.362760] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7668.296177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.712243] LustreError: 246390:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.799069] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7673.772780] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7682.355924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.203820] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7690.071814] Lustre: 247532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7697.380614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.857931] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7713.210154] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 7713.954091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7719.440407] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7720.507615] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7728.086495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7732.044630] Lustre: 249404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7739.887321] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.897340] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7739.899822] Lustre: Skipped 2 previous similar messages [ 7739.912758] Lustre: Skipped 7 previous similar messages [ 7742.073641] Lustre: server umount tfs-MDT0000 complete [ 7744.995655] LustreError: 246391:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.016794] LustreError: 246391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7746.224296] LustreError: 246368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391409 with bad export cookie 5058867184736239491 [ 7746.236919] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7746.244452] LustreError: 246368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7746.809972] Lustre: server umount tfs-MDT0001 complete [ 7761.766856] Lustre: server umount tfs-OST0000 complete [ 7765.472080] Lustre: 234916:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391412/real 1787391412] req@ffff94b895016a00 x1874215336872576/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787391428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7765.497952] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7765.513368] Lustre: Skipped 1 previous similar message [ 7765.870619] Lustre: server umount tfs-OST0001 complete [ 7792.660833] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 7805.467314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.185913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.200451] Lustre: Skipped 3 previous similar messages [ 7811.487789] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7822.364031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.342583] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7830.436763] Lustre: 252796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7838.499733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.713418] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7850.224749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 7857.914751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.805713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7866.279181] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 7874.929205] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7879.465940] Lustre: 254669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7887.847159] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 05:39:07 (1787391547) [ 7909.862025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.866261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7909.874621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7909.886760] Lustre: Skipped 3 previous similar messages [ 7909.913328] Lustre: Skipped 3 previous similar messages [ 7911.473064] Lustre: server umount lustre-MDT0000 complete [ 7914.985228] LustreError: 254432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7915.010860] LustreError: 254432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7915.779989] LustreError: 251630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391578 with bad export cookie 5058867184736241150 [ 7915.793482] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7915.800029] LustreError: 251630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7920.103413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7920.119032] Lustre: Skipped 1 previous similar message [ 7930.335126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7930.687892] Lustre: server umount lustre-MDT0001 complete [ 7949.279584] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7949.451652] Lustre: server umount lustre-OST0000 complete [ 7959.698445] Lustre: server umount lustre-OST0001 complete [ 7966.470505] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 7977.528569] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8018.033818] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8018.033818] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8018.033818] [ 8018.119423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8018.354858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8018.413634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8018.520487] Lustre: lustre-MDT0000: new disk, initializing [ 8018.718927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.728206] Lustre: Skipped 3 previous similar messages [ 8018.790705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8024.503809] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8037.193219] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8037.193219] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8037.193219] [ 8037.231765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8037.461794] Lustre: 257856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8037.595406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8037.615245] Lustre: Skipped 1 previous similar message [ 8037.798951] Lustre: lustre-MDT0001: new disk, initializing [ 8038.020936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8038.036755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8043.026094] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8053.945382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8054.176159] Lustre: lustre-OST0000: new disk, initializing [ 8054.179720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8054.185060] Lustre: 259441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8055.968178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8055.979647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8056.085715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8060.711295] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8071.723380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.924795] Lustre: lustre-OST0001: new disk, initializing [ 8071.929502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8071.937381] Lustre: 260312:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8073.490730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8073.512332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8073.674590] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8078.840613] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8089.964715] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 05:42:30 (1787391750) [ 8106.924096] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:42:47 (1787391767) [ 8122.850464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8122.859640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.891526] Lustre: Skipped 2 previous similar messages [ 8122.899730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8122.910197] Lustre: Skipped 5 previous similar messages [ 8128.590977] Lustre: server umount lustre-OST0000 complete [ 8135.136936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8135.148180] LustreError: Skipped 1 previous similar message [ 8138.970699] Lustre: server umount lustre-MDT0000 complete [ 8142.845579] LustreError: 260311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391805 with bad export cookie 5058867184736242711 [ 8142.854780] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8142.858573] LustreError: 260311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8149.357327] Lustre: server umount lustre-MDT0001 complete [ 8162.128117] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8174.106918] Lustre: server umount lustre-OST0001 complete [ 8181.096780] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8199.633348] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8240.795687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.301314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8241.333910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8241.437468] Lustre: lustre-MDT0000: new disk, initializing [ 8241.592693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8248.024632] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8260.806076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.019059] Lustre: 264870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8261.096520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8261.105244] Lustre: Skipped 1 previous similar message [ 8261.216476] Lustre: lustre-MDT0001: new disk, initializing [ 8261.475695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8261.490284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8267.183414] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8278.493417] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8285.263581] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8293.097194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8293.479658] Lustre: lustre-OST0000: new disk, initializing [ 8293.488210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8293.504777] Lustre: 266840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8293.672947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.684240] Lustre: Skipped 5 previous similar messages [ 8294.684958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8294.702664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8294.945327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8300.646051] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8311.345714] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8320.481876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8320.489772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.502572] LustreError: Skipped 1 previous similar message [ 8320.503945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.503951] Lustre: Skipped 7 previous similar messages [ 8320.540993] Lustre: Skipped 6 previous similar messages [ 8325.609129] LustreError: 266824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8325.626245] LustreError: 266824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 8325.835386] Lustre: server umount lustre-OST0000 complete [ 8344.031183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8344.353977] Lustre: server umount lustre-MDT0000 complete [ 8347.740322] LustreError: 264861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392010 with bad export cookie 5058867184736244181 [ 8347.746174] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8347.759084] LustreError: 264861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8348.067725] Lustre: server umount lustre-MDT0001 complete [ 8360.415230] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8364.218354] Key type lgssc unregistered [ 8364.744716] LNet: 268599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8364.754991] LNetError: 268599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8364.795392] LNet: Removed LNI 192.168.204.128@tcp [ 8366.171174] Key type .llcrypt unregistered [ 8366.176314] Key type ._llcrypt unregistered [ 8390.918494] Key type ._llcrypt registered [ 8390.928954] Key type .llcrypt registered [ 8391.074535] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8392.691825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8392.805931] alg: No test for adler32 (adler32-zlib) [ 8393.863695] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8394.137063] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8395.770407] Key type lgssc registered [ 8397.066502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8404.359776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8404.371985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.781965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.730597] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8415.222628] LustreError: 270063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.240391] LustreError: 270063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8417.034574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.404803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8421.405592] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8427.292206] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8432.943538] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8439.114400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.595588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8445.946565] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8450.876777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8453.746081] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8454.723376] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8458.210065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8458.218054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.232496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8458.568393] Lustre: server umount lustre-OST0000 complete [ 8463.329954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8463.343653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.370800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8466.401467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8466.402742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.521962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8471.528031] Lustre: Skipped 2 previous similar messages [ 8472.624877] Lustre: server umount lustre-MDT0000 complete [ 8476.387087] LustreError: 270045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392139 with bad export cookie 9564706299787932653 [ 8476.388145] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8476.404840] LustreError: 270045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8476.686484] Lustre: server umount lustre-MDT0001 complete [ 8487.813477] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8490.552080] Key type lgssc unregistered [ 8490.825561] LNet: 273649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8490.831344] LNetError: 273649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8490.855579] LNet: Removed LNI 192.168.204.128@tcp [ 8491.703245] Key type .llcrypt unregistered [ 8491.705465] Key type ._llcrypt unregistered [ 8509.367822] Key type ._llcrypt registered [ 8509.375768] Key type .llcrypt registered [ 8509.500599] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8510.576710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8510.612334] alg: No test for adler32 (adler32-zlib) [ 8511.707577] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8511.929232] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8513.591223] Key type lgssc registered [ 8514.946857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8524.835639] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8524.865152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8540.449879] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8545.567539] LustreError: 274846:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.128@tcp: failed processing log, type 4: rc = -110 [ 8572.255488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8578.871103] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8586.460284] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8588.513787] Lustre: server umount lustre-OST0000 complete [ 8606.022880] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 8608.888738] Key type lgssc unregistered [ 8609.142455] LNet: 276477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8609.153854] LNetError: 276477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8609.167408] LNet: Removed LNI 192.168.204.128@tcp [ 8609.990469] Key type .llcrypt unregistered [ 8609.996138] Key type ._llcrypt unregistered [ 8646.876735] Key type ._llcrypt registered [ 8646.883817] Key type .llcrypt registered [ 8647.006842] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8647.892700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8647.908307] alg: No test for adler32 (adler32-zlib) [ 8648.924752] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8649.144466] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 8650.823204] Key type lgssc registered [ 8651.886897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8663.903296] loop0: detected capacity change from 0 to 204800000 [ 8664.040328] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8664.049254] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8664.469174] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8665.499349] loop0: detected capacity change from 0 to 204800000 [ 8665.662089] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8665.673054] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8666.020686] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8667.207262] loop0: detected capacity change from 0 to 204800000 [ 8667.394738] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8667.406863] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8667.542738] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8668.606117] loop0: detected capacity change from 0 to 204800000 [ 8668.756980] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8670.063319] loop0: detected capacity change from 0 to 204800000 [ 8670.094531] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8674.625058] loop0: detected capacity change from 0 to 204800000 [ 8674.684320] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8678.596736] loop0: detected capacity change from 0 to 204800000 [ 8678.615983] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8682.253581] loop0: detected capacity change from 0 to 204800000 [ 8682.284716] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8686.061865] loop0: detected capacity change from 0 to 204800000 [ 8686.148514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8686.179559] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.305046] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8686.323498] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8686.341983] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8686.365990] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8686.384952] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8686.404771] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8686.430677] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8686.440287] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8686.451200] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8686.458659] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8686.466923] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8686.477893] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8686.486340] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8686.495577] Lustre: 279890:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8692.010957] Lustre: server umount MGS complete [ 8692.910383] loop0: detected capacity change from 0 to 204800000 [ 8692.976873] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.638168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8693.734473] Lustre: lustre-MDT0000: Aborting client recovery [ 8694.670321] loop1: detected capacity change from 0 to 204800000 [ 8694.814285] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8694.928561] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8694.937894] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8694.951193] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8694.959047] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8694.973627] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8694.983881] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8694.994782] Lustre: 280322:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8695.013875] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8695.153949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8695.212750] Lustre: lustre-MDT0001: Aborting client recovery [ 8696.216918] loop2: detected capacity change from 0 to 204800000 [ 8696.359900] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.452847] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8696.470795] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8696.496874] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8696.513682] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8696.532570] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8696.539650] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8696.551808] Lustre: 280405:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8696.687755] Lustre: Found index 0 for lustre-OST0000, updating log [ 8696.724560] Lustre: 280424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8696.796619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.819119] Lustre: lustre-OST0000: Aborting recovery [ 8697.737295] loop3: detected capacity change from 0 to 204800000 [ 8697.894164] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8697.984777] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8697.996726] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8698.009314] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8698.021780] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8698.035639] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8698.047450] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8698.060339] Lustre: 280490:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8698.077380] Lustre: Found index 1 for lustre-OST0001, updating log [ 8698.123342] Lustre: 280495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8698.197500] Lustre: lustre-OST0001: Aborting recovery [ 8703.340507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8703.354542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8703.381485] LustreError: 280266:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8703.396362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8703.402465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8704.361444] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8704.368853] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8705.916185] LustreError: 280233:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1205us: rc = -2 [ 8706.416960] LustreError: 280232:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 399us: rc = -2 [ 8706.435790] LustreError: 280232:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 19 previous similar messages [ 8719.841512] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.855708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.865121] Lustre: Skipped 2 previous similar messages [ 8719.880052] Lustre: Skipped 1 previous similar message [ 8724.962535] LustreError: 280349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.975354] LustreError: 280349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8725.056682] Lustre: server umount lustre-MDT0000 complete [ 8725.759657] LustreError: 280217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392388 with bad export cookie 7967963742836232373 [ 8725.789894] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8730.080682] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.082191] LustreError: 280231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.103558] Lustre: Skipped 1 previous similar message [ 8730.104576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8730.120303] LustreError: 280231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8730.150484] Lustre: Skipped 3 previous similar messages [ 8735.204963] LustreError: 280230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.205023] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8735.234520] Lustre: Skipped 1 previous similar message [ 8740.319201] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8740.643503] Lustre: server umount lustre-MDT0001 complete [ 8747.581936] Lustre: server umount lustre-OST0000 complete [ 8754.953068] Lustre: server umount lustre-OST0001 complete [ 8764.870417] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 05:53:45 (1787392425) [ 8792.712538] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 8804.905771] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 8847.983643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.233990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8848.259914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8848.347194] Lustre: lustre-MDT0000: new disk, initializing [ 8848.424332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.431968] Lustre: Skipped 1 previous similar message [ 8848.452380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8852.910679] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8864.595399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.782297] Lustre: 284568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8864.856437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8864.860574] Lustre: Skipped 1 previous similar message [ 8864.953833] Lustre: lustre-MDT0001: new disk, initializing [ 8865.094234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8865.161543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8865.173879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8869.837968] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8878.523091] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8884.500440] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8892.071729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.530599] Lustre: lustre-OST0000: new disk, initializing [ 8892.535457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8892.545837] Lustre: 286538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8892.688643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.827921] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 8902.742091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8902.754532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8902.943503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8909.034277] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8914.075760] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8925.812904] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8932.886553] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8944.738656] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8985.062033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8985.064240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8985.077382] LustreError: Skipped 1 previous similar message [ 8985.078578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8985.078584] Lustre: Skipped 2 previous similar messages [ 8985.114155] Lustre: Skipped 2 previous similar messages [ 8987.313150] Lustre: server umount lustre-OST0000 complete [ 8993.249829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8993.253799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8993.267421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8993.273765] Lustre: Skipped 1 previous similar message [ 8996.805615] Lustre: server umount lustre-MDT0000 complete [ 9000.264723] LustreError: 284558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392663 with bad export cookie 7967963742836235082 [ 9000.273458] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.281199] LustreError: 284558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9000.774089] Lustre: server umount lustre-MDT0001 complete [ 9009.946844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.254695] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9020.332667] Lustre: server umount MGS complete [ 9033.422460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.097260] LustreError: 290482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.133320] LustreError: 290482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9034.171352] LustreError: 290495:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9034.308664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9039.331459] LustreError: 290483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9040.122493] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9048.551574] LustreError: 290483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9048.579594] LustreError: 290483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9048.649352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.969269] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9061.003691] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9067.094922] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9072.165765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9072.511649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.518784] Lustre: Skipped 1 previous similar message [ 9077.475167] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9083.665366] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9109.473407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9109.481055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9109.493514] Lustre: Skipped 1 previous similar message [ 9109.500020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9109.504618] Lustre: Skipped 3 previous similar messages [ 9114.776668] Lustre: server umount lustre-OST0000 complete [ 9119.714540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9119.729696] Lustre: Skipped 2 previous similar messages [ 9124.596683] Lustre: server umount lustre-MDT0000 complete [ 9124.833715] LustreError: 290482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.840102] LustreError: 290482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9128.022119] LustreError: 290464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392790 with bad export cookie 7967963742836237007 [ 9128.033679] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9128.039259] LustreError: 290464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9128.415871] Lustre: server umount lustre-MDT0001 complete [ 9139.078611] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9141.758921] Key type lgssc unregistered [ 9142.069879] LNet: 294425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9142.080118] LNetError: 294425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9142.097600] LNet: Removed LNI 192.168.204.128@tcp [ 9142.871515] Key type .llcrypt unregistered [ 9142.876358] Key type ._llcrypt unregistered [ 9179.646321] Key type ._llcrypt registered [ 9179.649412] Key type .llcrypt registered [ 9179.786331] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9195.216161] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9195.925886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9196.036496] alg: No test for adler32 (adler32-zlib) [ 9197.114952] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9197.341320] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9199.015172] Key type lgssc registered [ 9200.066550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9233.799878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9233.811347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.121492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9235.147772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9235.246379] Lustre: lustre-MDT0000: new disk, initializing [ 9235.313942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9235.338920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9239.531491] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9249.195871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.328186] Lustre: 298395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9249.405521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9249.410047] Lustre: Skipped 1 previous similar message [ 9249.486625] Lustre: lustre-MDT0001: new disk, initializing [ 9249.598704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9249.666143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9249.677773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9254.290195] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9262.886131] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9268.134681] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9274.133366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9274.322798] Lustre: lustre-OST0000: new disk, initializing [ 9274.326176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9274.332442] Lustre: 300362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9274.396582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9280.414653] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9281.609029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9281.630640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9281.729870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9288.642142] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9296.526408] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9305.933163] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9312.132407] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9316.632134] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9363.941148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.944317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9363.965403] Lustre: Skipped 1 previous similar message [ 9363.966321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9363.988304] LustreError: Skipped 1 previous similar message [ 9369.068626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.073551] Lustre: Skipped 2 previous similar messages [ 9374.178747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9374.189204] Lustre: Skipped 1 previous similar message [ 9377.247164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9377.521762] Lustre: server umount lustre-OST0000 complete [ 9382.889211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9382.898726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.926070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9384.418854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9387.719591] Lustre: server umount lustre-MDT0000 complete [ 9391.590239] LustreError: 300361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393054 with bad export cookie 5229309854854498974 [ 9391.600314] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9391.608293] LustreError: 300361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9392.084947] Lustre: server umount lustre-MDT0001 complete [ 9401.304459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.216163] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9412.236966] Lustre: server umount MGS complete [ 9422.519677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.809334] LustreError: 304185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9422.832759] LustreError: 304185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9422.839655] LustreError: 304198:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9422.899040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9426.815799] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9427.946686] LustreError: 304186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9433.061308] LustreError: 304185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.048350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9435.369494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9439.169497] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9446.175469] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9451.321151] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9457.196108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9457.653263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.510461] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9472.742719] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9496.547413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.554593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.581571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.588304] Lustre: Skipped 2 previous similar messages [ 9498.592322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.362626] Lustre: server umount lustre-OST0000 complete [ 9507.297429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9507.325962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9507.340529] Lustre: Skipped 1 previous similar message [ 9507.355112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9507.366901] Lustre: Skipped 2 previous similar messages [ 9512.650305] Lustre: server umount lustre-MDT0000 complete [ 9513.955213] LustreError: 304186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9513.988354] LustreError: 304186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9516.295396] LustreError: 304166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393179 with bad export cookie 5229309854854500829 [ 9516.297598] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.312809] LustreError: 304166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9516.670486] Lustre: server umount lustre-MDT0001 complete [ 9527.402536] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [ 9530.412911] Key type lgssc unregistered [ 9530.869493] LNet: 308129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9530.880456] LNetError: 308129:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9530.901442] LNet: Removed LNI 192.168.204.128@tcp [ 9531.897510] Key type .llcrypt unregistered [ 9531.903994] Key type ._llcrypt unregistered [ 9551.905996] Key type ._llcrypt registered [ 9551.908250] Key type .llcrypt registered [ 9552.030885] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [ 9566.318662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9567.565129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9567.625248] alg: No test for adler32 (adler32-zlib) [ 9568.803115] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9569.133791] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [ 9570.848846] Key type lgssc registered [ 9572.219546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9598.537660] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [ 9613.117339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9613.139557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.388790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9614.442827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9614.509247] Lustre: lustre-MDT0000: new disk, initializing [ 9614.558767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.574156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9618.768260] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [ 9628.899474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9629.163508] Lustre: lustre-OST0000: new disk, initializing [ 9629.169953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9629.174179] Lustre: Skipped 1 previous similar message [ 9629.178305] Lustre: 312178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9629.241788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9634.326394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9634.336401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9634.360043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9634.967498] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10477.031874] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.044987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10481.110575] Lustre: server umount lustre-MDT0000 complete [10485.022967] LustreError: 313156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394147 with bad export cookie 7600612298748765102 [10485.027337] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10485.041342] LustreError: 313156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10485.240755] Lustre: server umount lustre-OST0000 complete [10513.445088] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [10524.334753] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10561.339843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.604705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10561.630965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10561.715528] Lustre: lustre-MDT0000: new disk, initializing [10561.785872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10561.799257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10565.925434] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10576.917826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10577.119109] Lustre: 316529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10577.197531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10577.204942] Lustre: Skipped 1 previous similar message [10577.305747] Lustre: lustre-MDT0001: new disk, initializing [10577.422504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10577.473985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10577.483627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10582.905423] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10592.186334] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10598.450187] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10605.314784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10605.588440] Lustre: lustre-OST0000: new disk, initializing [10605.592938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10605.600838] Lustre: 318498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10605.704287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10611.525800] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10613.900221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10613.916339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10614.050849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10620.711488] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10625.622411] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10625.922935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10631.317483] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10631.609832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10634.723409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.725623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10634.737251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10634.743802] LustreError: Skipped 1 previous similar message [10634.756676] Lustre: Skipped 1 previous similar message [10639.905844] Lustre: server umount lustre-OST0000 complete [10643.937566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.950772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.965773] Lustre: Skipped 1 previous similar message [10643.979633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10650.086328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10650.091321] Lustre: Skipped 3 previous similar messages [10654.180422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10654.189112] Lustre: Skipped 1 previous similar message [10658.271190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10658.552400] Lustre: server umount lustre-MDT0000 complete [10659.296306] LustreError: 316542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.318216] LustreError: 316542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10662.689201] LustreError: 316523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394325 with bad export cookie 7600612298751705921 [10662.695788] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10662.709489] LustreError: 316523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10663.184656] Lustre: server umount lustre-MDT0001 complete [10666.888508] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10668.988551] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 06:25:29 (1787394329) [10676.417097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.921091] LustreError: 320813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10677.000129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10681.745200] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10682.338342] LustreError: 320814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.435048] LustreError: 320813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.523578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.979064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10694.571358] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10701.189631] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10706.718762] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10712.950354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10713.407359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10719.408498] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10726.174568] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10732.899177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10733.100209] Lustre: lustre-OST0001: new disk, initializing [10733.104445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10733.109912] Lustre: 323648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10733.226750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10739.103897] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10741.946223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10741.981880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10742.115862] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10742.143798] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10743.147732] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10744.227314] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10744.239947] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10746.277770] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10746.286026] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10748.834359] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10749.198693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10750.497967] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10750.515331] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10757.262203] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fd5911a2800.ost_server_uuid 50 [10758.827250] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10758.860470] LustreError: 324121:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10760.368258] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9fd5911a2800.ost_server_uuid in FULL state after 0 sec [10767.329524] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fd5911a2800.ost_server_uuid 50 [10768.696466] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fd5911a2800.ost_server_uuid in IDLE state after 0 sec [10770.238664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10785.160179] Lustre: lustre-OST0001: Not available for connect from 192.168.204.28@tcp (stopping) [10785.760946] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10785.764456] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10785.776490] LustreError: Skipped 1 previous similar message [10785.797133] Lustre: Skipped 1 previous similar message [10790.491623] Lustre: server umount lustre-OST0001 complete [10793.450281] LustreError: 322633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.485077] LustreError: 322633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10798.561365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10798.570701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10798.609264] Lustre: Skipped 2 previous similar messages [10803.111941] Lustre: server umount lustre-OST0000 complete [10807.777182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10807.783496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10807.793392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10807.797619] Lustre: Skipped 7 previous similar messages [10813.595968] Lustre: server umount lustre-MDT0000 complete [10813.920920] LustreError: 320832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.937815] LustreError: 320832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10817.305054] LustreError: 323647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394480 with bad export cookie 7600612298751706803 [10817.309244] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10817.314394] LustreError: 323647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10817.588341] Lustre: server umount lustre-MDT0001 complete [10827.296641] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [10829.708838] Key type lgssc unregistered [10829.995023] LNet: 326334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10830.003950] LNetError: 326334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10830.020489] LNet: Removed LNI 192.168.204.128@tcp [10830.826823] Key type .llcrypt unregistered [10830.834864] Key type ._llcrypt unregistered [10857.063466] Key type ._llcrypt registered [10857.065357] Key type .llcrypt registered [10857.167589] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [10858.242885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10858.358682] alg: No test for adler32 (adler32-zlib) [10859.439273] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [10859.621707] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [10861.313704] Key type lgssc registered [10862.802532] Lustre: Echo OBD driver; http://www.lustre.org/ [10871.220973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10871.251639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.772520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10877.261916] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10883.068688] LustreError: 327759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.109328] LustreError: 327759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10885.332296] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10888.163318] LustreError: 327758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.374797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.820958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10897.046851] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10905.763081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10906.180768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10912.183623] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10917.422736] LustreError: 330063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.459914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10919.017277] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10923.493099] LustreError: 329442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.511717] LustreError: 329442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10925.621360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.821281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10931.195765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10931.375065] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [10939.949573] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10947.092501] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10947.271493] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11048.941289] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11048.950211] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.967786] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11048.981319] Lustre: Skipped 1 previous similar message [11052.870929] Lustre: server umount lustre-OST0001 complete [11054.049747] LustreError: 330063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.074219] LustreError: 330063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11059.177572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.181842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11059.196373] Lustre: Skipped 2 previous similar messages [11059.210645] Lustre: Skipped 1 previous similar message [11064.321868] Lustre: server umount lustre-OST0000 complete [11069.411550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11069.422639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11069.426077] Lustre: Skipped 1 previous similar message [11074.531633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11074.537657] Lustre: Skipped 1 previous similar message [11079.652897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11079.663230] Lustre: Skipped 1 previous similar message [11082.208258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11082.456680] Lustre: server umount lustre-MDT0000 complete [11084.777140] LustreError: 327759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.800284] LustreError: 327759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11085.750719] LustreError: 331447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394748 with bad export cookie 12441663541122160812 [11085.754502] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11085.765022] LustreError: 331447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11086.064761] Lustre: server umount lustre-MDT0001 complete [11096.304049] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11099.054417] Key type lgssc unregistered [11099.294147] LNet: 332704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11099.305236] LNetError: 332704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11099.330099] LNet: Removed LNI 192.168.204.128@tcp [11100.104511] Key type .llcrypt unregistered [11100.113915] Key type ._llcrypt unregistered [11144.715369] Key type ._llcrypt registered [11144.717654] Key type .llcrypt registered [11144.827228] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11145.990447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11146.143991] alg: No test for adler32 (adler32-zlib) [11147.204246] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11147.453802] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11149.136115] Key type lgssc registered [11150.231332] Lustre: Echo OBD driver; http://www.lustre.org/ [11160.516202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11160.530868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11162.125278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11166.102718] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11171.302302] LustreError: 334994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.319434] LustreError: 334994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11173.789898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11174.260787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11178.237647] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11181.060824] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11186.900349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11187.314973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11193.243841] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11198.578033] LustreError: 336486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11200.377791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11200.665707] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11206.619217] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11207.153471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11207.160471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11214.747357] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11222.647824] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11232.738320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11232.751693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11232.757523] Lustre: Skipped 3 previous similar messages [11232.762840] Lustre: Skipped 2 previous similar messages [11237.861661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11237.874787] Lustre: Skipped 4 previous similar messages [11238.549152] Lustre: server umount lustre-MDT0000 complete [11242.031481] LustreError: 334975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394904 with bad export cookie 9600081614722477998 [11242.040788] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11242.045078] LustreError: 334975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11242.392332] Lustre: server umount lustre-MDT0001 complete [11257.260625] Lustre: server umount lustre-OST0000 complete [11258.336140] Lustre: 334258:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394905/real 1787394905] req@ffff94b88e9e2300 x1874219300352128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11258.393319] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.254841] Lustre: server umount lustre-OST0001 complete [11277.538217] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11288.335534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.712273] LustreError: 340260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.779690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11293.479084] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11294.181926] LustreError: 340261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.273703] LustreError: 340260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.053629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11302.526178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11306.896205] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11309.655441] Lustre: 341402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11315.844886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.642191] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11327.545457] LustreError: 341757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.290990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11331.582441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11331.591640] Lustre: Skipped 1 previous similar message [11333.632278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11333.640398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11337.603576] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11345.252507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11348.651427] Lustre: 343274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11353.572460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11353.582362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.601187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11358.476124] Lustre: server umount lustre-MDT0000 complete [11359.206172] LustreError: 340260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11359.249736] LustreError: 340260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11361.613730] LustreError: 341002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395024 with bad export cookie 9600081614722479699 [11361.635529] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11361.928635] Lustre: server umount lustre-MDT0001 complete [11376.255711] Lustre: server umount lustre-OST0000 complete [11389.832454] Lustre: server umount lustre-OST0001 complete [11393.949940] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11395.453838] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11397.404596] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 06:37:37 (1787395057) [11420.699744] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11429.184437] loop0: detected capacity change from 0 to 1099511627776 [11429.855827] SGI XFS with ACLs, security attributes, quota, no debug enabled [11429.874551] *** VALIDATE xfs *** [11429.902976] XFS (loop0): Mounting V5 Filesystem [11429.949582] XFS (loop0): Ending clean mount [11431.161528] loop1: detected capacity change from 0 to 18691697672192 [11538.710676] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [11548.148300] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11583.628221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.886845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11583.950786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11584.100874] Lustre: lustre-MDT0000: new disk, initializing [11584.169339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11584.196690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11589.095662] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11600.298443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11600.528799] Lustre: 348407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11600.627822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11600.639214] Lustre: Skipped 1 previous similar message [11600.747330] Lustre: lustre-MDT0001: new disk, initializing [11600.902889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11600.916166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11605.782566] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11615.102617] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11621.209604] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11628.834423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11629.120395] Lustre: lustre-OST0000: new disk, initializing [11629.125071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11629.133452] Lustre: 350379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11630.980470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11630.994628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11631.114259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11637.254555] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11646.768470] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11651.853541] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11652.047459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11655.811655] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11656.014490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11661.798953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.801319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11661.803571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11661.803579] Lustre: Skipped 4 previous similar messages [11661.821970] Lustre: Skipped 4 previous similar messages [11661.826416] LustreError: Skipped 1 previous similar message [11663.836896] Lustre: server umount lustre-OST0000 complete [11666.915533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11666.925659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11666.931371] Lustre: Skipped 1 previous similar message [11666.941399] Lustre: Skipped 1 previous similar message [11677.153322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11677.170807] Lustre: Skipped 3 previous similar messages [11681.247146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11681.470792] Lustre: server umount lustre-MDT0000 complete [11685.231764] LustreError: 350378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395348 with bad export cookie 9600081614722481246 [11685.232503] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11685.245302] LustreError: 350378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11685.571453] Lustre: server umount lustre-MDT0001 complete [11693.139690] XFS (loop0): Unmounting Filesystem [11698.399390] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 06:42:38 (1787395358) [11705.955757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.400991] LustreError: 352940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.488934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11706.496290] Lustre: Skipped 2 previous similar messages [11710.759650] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11719.423370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.025763] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11733.271453] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11738.852825] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11745.619750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.243623] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11759.246617] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11778.017471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11778.032532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11778.052814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11778.061506] Lustre: Skipped 2 previous similar messages [11781.089117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11783.735240] Lustre: server umount lustre-OST0000 complete [11791.329125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11794.770254] Lustre: server umount lustre-MDT0000 complete [11796.449572] LustreError: 355456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11796.474502] LustreError: 355456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11798.716202] LustreError: 352920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395461 with bad export cookie 9600081614722482135 [11798.740174] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11798.754983] LustreError: 352920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11799.123292] Lustre: server umount lustre-MDT0001 complete [11810.561335] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [11813.813508] Key type lgssc unregistered [11814.151329] LNet: 356593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11814.167499] LNetError: 356593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11814.190443] LNet: Removed LNI 192.168.204.128@tcp [11815.118179] Key type .llcrypt unregistered [11815.119669] Key type ._llcrypt unregistered [11842.451937] Key type ._llcrypt registered [11842.453414] Key type .llcrypt registered [11842.568198] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [11843.614828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11843.626522] alg: No test for adler32 (adler32-zlib) [11844.879756] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11845.266451] LNet: Added LNI 192.168.204.128@tcp [8/256/0/180] [11846.951317] Key type lgssc registered [11848.109732] Lustre: Echo OBD driver; http://www.lustre.org/ [11856.192168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11856.214300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.678871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.366964] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11868.136060] LustreError: 358158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.147796] LustreError: 358158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11870.970694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.398530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11876.298919] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11883.589510] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11889.297298] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11895.569660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11895.933686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11902.152526] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11908.431897] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11916.160109] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [11917.794411] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11917.804965] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11917.825220] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11918.829924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11918.844357] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11921.246791] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [11921.256530] Lustre: Skipped 1 previous similar message [11921.893526] Lustre: server umount lustre-MDT0001 complete [11923.942346] LustreError: 358886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.964088] LustreError: 358886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11926.352474] LustreError: 358154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.063572] LustreError: 358158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.095274] LustreError: 358158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11931.740434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.839509] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11931.842665] Lustre: Found index 1 for lustre-MDT0001, updating log [11931.891623] Lustre: 358148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11932.234767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11936.322475] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [11937.254879] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11941.886837] LustreError: 361760:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11941.898584] LustreError: 361760:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11941.918619] LustreError: 358150:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11941.987328] LustreError: 361763:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -17 [11941.999333] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11941.999333] [12243.820262] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12244.461659] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12244.465934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.477901] Lustre: Skipped 1 previous similar message [12246.963204] Lustre: server umount lustre-MDT0001 complete [12248.930701] LustreError: 359842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.961089] LustreError: 359842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12258.319730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.479727] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12258.489974] Lustre: Found index 1 for lustre-MDT0001, updating log [12258.576326] Lustre: 358148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12258.934809] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12264.400981] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12264.466109] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12267.531863] LustreError: 370384:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12267.538053] LustreError: 370384:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12267.544638] LustreError: 358150:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12267.551965] LustreError: 358150:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12267.627117] LustreError: 370387:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12267.631317] LustreError: 370387:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12267.637552] LustreError: 370387:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -17 [12267.649426] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.128@tcp [12267.649426] [12571.502501] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.28@tcp (stopping) [12571.517296] Lustre: Skipped 2 previous similar messages [12571.623487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12571.630497] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12571.649318] Lustre: Skipped 1 previous similar message [12576.283767] Lustre: server umount lustre-MDT0001 complete [12576.599751] LustreError: 358153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.629351] LustreError: 358153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12587.671990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12587.904523] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12587.908296] Lustre: Found index 1 for lustre-MDT0001, updating log [12587.957797] Lustre: 358146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12588.352783] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12593.263218] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [12593.653411] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12597.766168] LustreError: 378856:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12597.778675] LustreError: 378856:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12597.785755] LustreError: 358150:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12597.790196] LustreError: 358150:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12597.843898] LustreError: 378859:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12597.865114] LustreError: 378859:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.128@tcp: cfg command failed: rc = -114 [12597.876364] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12597.876364] [12916.192531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12916.194789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.211491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12916.236602] Lustre: Skipped 1 previous similar message [12916.243884] Lustre: Skipped 2 previous similar messages [12920.495467] Lustre: server umount lustre-MDT0000 complete [12921.312689] LustreError: 358153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.346703] LustreError: 358153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [12924.245102] LustreError: 358139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396587 with bad export cookie 13807953811977037322 [12924.251771] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12926.437691] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12926.455222] Lustre: Skipped 1 previous similar message [12930.811880] Lustre: server umount lustre-MDT0001 complete [12934.907931] Lustre: server umount lustre-OST0000 complete [12944.209828] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [12954.363867] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [12989.867482] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 07:04:10 (1787396650) [12996.513833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12996.764973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12996.802942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12996.872485] Lustre: lustre-MDT0000: new disk, initializing [12996.947179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12996.984179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13001.559788] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13011.402897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13011.513908] Lustre: 389348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13011.552809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13011.556090] Lustre: Skipped 1 previous similar message [13011.611947] Lustre: lustre-MDT0001: new disk, initializing [13011.690584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13011.729147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13011.741271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13015.958529] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13024.677947] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13029.468801] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13035.829434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13036.062704] Lustre: lustre-OST0000: new disk, initializing [13036.071063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13036.082224] Lustre: 391313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13036.208822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13042.776608] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_default_debug -1 all [13045.903542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13045.913514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13046.108669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13053.574848] Lustre: DEBUG MARKER: oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13061.608913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.610912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13061.629631] Lustre: Skipped 2 previous similar messages [13061.644214] Lustre: Skipped 6 previous similar messages [13063.599280] Lustre: server umount lustre-MDT0000 complete [13066.722936] LustreError: 389359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.757892] LustreError: 389359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13067.905499] LustreError: 389340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396730 with bad export cookie 13807953811977039275 [13067.913983] LustreError: MGC192.168.204.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13067.914278] LustreError: 389340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13067.915569] LustreError: 390153:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13067.915719] LustreError: 390153:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b43d5506 x1874220032585216/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13068.556174] Lustre: server umount lustre-MDT0001 complete [13082.892133] Lustre: server umount lustre-OST0000 complete [13110.920095] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing set_hostid [13121.964160] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing load_modules_local [13152.158871] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12853 sec ========= 07:06:52 (1787396812) [13154.181179] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:06:54 (1787396814) === [13157.489907] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:06:57 (1787396817) === [13191.731404] Lustre: DEBUG MARKER: oleg428-server.virtnet: executing unload_modules_local [13194.515705] Key type lgssc unregistered [13194.794649] LNet: 397680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13194.804055] LNetError: 397680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13194.816970] LNet: Removed LNI 192.168.204.128@tcp [13195.559665] Key type .llcrypt unregistered [13195.561966] Key type ._llcrypt unregistered