[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527437249 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003327] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007948] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011067] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.022151] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026820] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030147] *** VALIDATE tmpfs *** [ 0.032203] *** VALIDATE proc *** [ 0.033278] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.035301] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042797] debug: unmapping init [mem 0xffffffff9fe59000-0xffffffff9fe60fff] [ 0.044985] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045770] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046043] ... version: 2 [ 0.047018] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050021] ... max period: 00007fffffffffff [ 0.051029] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.054284] rcu: Hierarchical SRCU implementation. [ 0.056600] smp: Bringing up secondary CPUs ... [ 0.057668] x86: Booting SMP configuration: [ 0.058036] .... node #0, CPUs: #1 #2 #3 [ 0.061782] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.190845] node 0 deferred pages initialised in 123ms [ 0.192130] devtmpfs: initialized [ 0.193299] x86/mm: Memory block size: 128MB [ 0.196386] gcov: version magic: 0x41383552 [ 0.198335] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.199142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.200358] pinctrl core: initialized pinctrl subsystem [ 0.201249] [ 0.201894] ************************************************************* [ 0.202027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203019] ** ** [ 0.204020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.205023] ** ** [ 0.206028] ** This means that this kernel is built to expose internal ** [ 0.207020] ** IOMMU data structures, which may compromise security on ** [ 0.208020] ** your system. ** [ 0.209019] ** ** [ 0.210018] ** If you see this message and you are not debugging the ** [ 0.211021] ** kernel, report this immediately to your vendor! ** [ 0.212020] ** ** [ 0.213018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214017] ************************************************************* [ 0.215728] NET: Registered protocol family 16 [ 0.216474] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.220069] cpuidle: using governor menu [ 0.221995] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.224526] PCI: Using configuration type 1 for base access [ 0.226159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.234176] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236189] cryptd: max_cpu_qlen set to 1000 [ 0.238018] ACPI: Added _OSI(Module Device) [ 0.239017] ACPI: Added _OSI(Processor Device) [ 0.240018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245135] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248581] ACPI: Interpreter enabled [ 0.249078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250019] ACPI: Using IOAPIC for interrupt routing [ 0.251165] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261310] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266370] acpiphp: Slot [2] registered [ 0.267182] acpiphp: Slot [5] registered [ 0.268129] acpiphp: Slot [6] registered [ 0.269120] acpiphp: Slot [7] registered [ 0.270116] acpiphp: Slot [8] registered [ 0.271109] acpiphp: Slot [9] registered [ 0.272112] acpiphp: Slot [10] registered [ 0.273147] acpiphp: Slot [3] registered [ 0.274094] acpiphp: Slot [4] registered [ 0.275098] acpiphp: Slot [11] registered [ 0.276081] acpiphp: Slot [12] registered [ 0.277080] acpiphp: Slot [13] registered [ 0.278080] acpiphp: Slot [14] registered [ 0.279076] acpiphp: Slot [15] registered [ 0.280101] acpiphp: Slot [16] registered [ 0.281098] acpiphp: Slot [17] registered [ 0.282082] acpiphp: Slot [18] registered [ 0.283081] acpiphp: Slot [19] registered [ 0.284076] acpiphp: Slot [20] registered [ 0.285082] acpiphp: Slot [21] registered [ 0.286116] acpiphp: Slot [22] registered [ 0.287095] acpiphp: Slot [23] registered [ 0.288076] acpiphp: Slot [24] registered [ 0.289079] acpiphp: Slot [25] registered [ 0.290074] acpiphp: Slot [26] registered [ 0.291083] acpiphp: Slot [27] registered [ 0.292112] acpiphp: Slot [28] registered [ 0.293100] acpiphp: Slot [29] registered [ 0.294081] acpiphp: Slot [30] registered [ 0.295070] acpiphp: Slot [31] registered [ 0.296068] PCI host bridge to bus 0000:00 [ 0.297018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304883] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306148] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.319521] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.320677] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.321053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322750] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.334015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.336756] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.341217] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.356020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.395021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414410] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.443021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.452511] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.458023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.462023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.474019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.483735] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.497031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.531031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.541172] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.548023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.556022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.588495] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.596023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.604019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.626022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.639000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.641439] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644414] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.647158] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.649403] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.654038] iommu: Default domain type: Passthrough [ 0.656433] SCSI subsystem initialized [ 0.657138] ACPI: bus type USB registered [ 0.658094] usbcore: registered new interface driver usbfs [ 0.660082] usbcore: registered new interface driver hub [ 0.662102] usbcore: registered new device driver usb [ 0.663217] pps_core: LinuxPPS API ver. 1 registered [ 0.665017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.668071] PTP clock support registered [ 0.669165] EDAC MC: Ver: 3.0.0 [ 0.671039] PCI: Using ACPI for IRQ routing [ 0.673094] NetLabel: Initializing [ 0.675014] NetLabel: domain hash size = 128 [ 0.677017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.680109] NetLabel: unlabeled traffic allowed by default [ 0.683175] vgaarb: loaded [ 0.685328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.687014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.693629] clocksource: Switched to clocksource kvm-clock [ 0.787240] VFS: Disk quotas dquot_6.6.0 [ 0.788262] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.790215] *** VALIDATE ramfs *** [ 0.790886] *** VALIDATE hugetlbfs *** [ 0.792137] pnp: PnP ACPI init [ 0.794359] pnp: PnP ACPI: found 6 devices [ 0.811904] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.814910] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.816294] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.818521] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.820713] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.823218] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.825251] NET: Registered protocol family 2 [ 0.827718] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.833052] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.836227] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841361] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.843618] TCP: Hash tables configured (established 65536 bind 65536) [ 0.846213] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.849608] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852569] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.855539] NET: Registered protocol family 1 [ 0.858055] RPC: Registered named UNIX socket transport module. [ 0.860310] RPC: Registered udp transport module. [ 0.861641] RPC: Registered tcp transport module. [ 0.863132] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.865155] NET: Registered protocol family 44 [ 0.866611] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.868329] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.870313] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.872513] PCI: CLS 0 bytes, default 64 [ 0.874925] Unpacking initramfs... [ 2.233582] debug: unmapping init [mem 0xffff94257cc54000-0xffff94257ffbffff] [ 2.237470] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.239646] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.242328] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.729709] Initialise system trusted keyrings [ 2.731647] Key type blacklist registered [ 2.734346] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.747139] zbud: loaded [ 2.751214] *** VALIDATE nfs *** [ 2.752672] *** VALIDATE nfs4 *** [ 2.754652] pstore: using deflate compression [ 2.758330] Platform Keyring initialized [ 2.857278] NET: Registered protocol family 38 [ 2.859883] Key type asymmetric registered [ 2.861781] Asymmetric key parser 'x509' registered [ 2.863912] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.867215] io scheduler mq-deadline registered [ 2.868377] io scheduler kyber registered [ 2.869452] io scheduler bfq registered [ 2.870949] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.872893] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.874769] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.876778] ACPI: Power Button [PWRF] [ 2.881323] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.886321] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.897823] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.904865] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.918217] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.949595] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.982784] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.988511] Non-volatile memory driver v1.3 [ 2.990120] Linux agpgart interface v0.103 [ 3.020125] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.024502] vda: detected capacity change from 0 to 76681216 [ 3.038606] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.040960] vdb: detected capacity change from 0 to 1073741824 [ 3.056394] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.058380] vdc: detected capacity change from 0 to 2621440000 [ 3.070531] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.073103] vdd: detected capacity change from 0 to 2621440000 [ 3.086327] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.088718] vde: detected capacity change from 0 to 4294967296 [ 3.101438] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.103960] vdf: detected capacity change from 0 to 4294967296 [ 3.114193] libphy: Fixed MDIO Bus: probed [ 3.119819] usbcore: registered new interface driver usbserial_generic [ 3.121673] usbserial: USB Serial support registered for generic [ 3.123964] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.127685] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.129370] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.131788] mousedev: PS/2 mouse device common for all mice [ 3.134281] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.136694] rtc_cmos 00:05: RTC can wake from S4 [ 3.141861] rtc_cmos 00:05: registered as rtc0 [ 3.143481] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.144243] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.144287] intel_pstate: CPU model not supported [ 3.150321] hid: raw HID events driver (C) Jiri Kosina [ 3.153725] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.157612] usbcore: registered new interface driver usbhid [ 3.163353] usbhid: USB HID core driver [ 3.164511] drop_monitor: Initializing network drop monitor service [ 3.166385] Initializing XFRM netlink socket [ 3.167907] NET: Registered protocol family 10 [ 3.170922] Segment Routing with IPv6 [ 3.172413] NET: Registered protocol family 17 [ 3.174456] mpls_gso: MPLS GSO support [ 3.179964] RAS: Correctable Errors collector initialized. [ 3.181645] AVX version of gcm_enc/dec engaged. [ 3.183182] AES CTR mode by8 optimization enabled [ 3.240345] sched_clock: Marking stable (3240283796, 0)->(4246154106, -1005870310) [ 3.243168] registered taskstats version 1 [ 3.244528] Loading compiled-in X.509 certificates [ 3.246843] zswap: loaded using pool lzo/zbud [ 3.263775] Key type big_key registered [ 3.272238] Key type encrypted registered [ 3.273245] ima: No TPM chip found, activating TPM-bypass! [ 3.274323] ima: Allocated hash algorithm: sha1 [ 3.275257] ima: No architecture policies found [ 3.276314] evm: Initialising EVM extended attributes: [ 3.277437] evm: security.selinux [ 3.278180] evm: security.ima [ 3.278893] evm: security.capability [ 3.279663] evm: HMAC attrs: 0x1 [ 3.281956] rtc_cmos 00:05: setting system clock to 2026-09-05 23:46:12 UTC (1788651972) [ 3.287086] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.289095] debug: unmapping init [mem 0xffffffff9fb82000-0xffffffff9fe58fff] [ 3.296189] Write protecting the kernel read-only data: 28672k [ 3.298147] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 3.299657] debug: unmapping init [mem 0xffffffff9eb14000-0xffffffff9ebfffff] [ 3.325766] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.331488] systemd[1]: Detected virtualization kvm. [ 3.332785] systemd[1]: Detected architecture x86-64. [ 3.333877] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.361992] systemd[1]: No hostname configured. [ 3.364092] systemd[1]: Set hostname to . [ 3.366585] random: systemd: uninitialized urandom read (16 bytes read) [ 3.369586] systemd[1]: Initializing machine ID from random generator. [ 3.437394] random: ln: uninitialized urandom read (6 bytes read) [ 3.505179] random: systemd: uninitialized urandom read (16 bytes read) [ 3.507446] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.512289] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.516513] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.054438] device-mapper: uevent: version 1.0.3 [ 4.055924] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK [ 4.655702] random: fast init done ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.689834] virtio_net virtio0 ens2: renamed from eth0 [ 4.704782] scsi host0: ata_piix [ 4.748246] scsi host1: ata_piix [ 4.750351] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.753049] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.974622] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.785394] random: crng init done [ 9.786873] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.053620] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.227495] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.567101] SELinux: Disabled at runtime. [ 11.653977] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.667553] systemd[1]: Detected virtualization kvm. [ 11.670749] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.819904] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.824322] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.832174] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.839826] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.845424] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.856474] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.882845] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ 13.111717] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.323285] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.677471] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.725723] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.157850] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.297638] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 21.019590] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 22.009470] NFS: Registering the id_resolver key type [ 22.018450] Key type id_resolver registered [ 22.020885] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg157-server login: [ 64.018603] hrtimer: interrupt took 4045145 ns [ 84.946514] libcfs: loading out-of-tree module taints kernel. [ 84.996434] Key type ._llcrypt registered [ 84.998159] Key type .llcrypt registered [ 85.083528] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 103.152696] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 105.377344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.391131] alg: No test for adler32 (adler32-zlib) [ 107.201703] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 108.494365] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 110.354248] Key type lgssc registered [ 113.204632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.442152] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.168204] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 184.002061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.066469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.394265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.456740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.513435] Lustre: lustre-MDT0000: new disk, initializing [ 185.588126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.618310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.560319] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.923849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.038431] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.061518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.065351] Lustre: Skipped 1 previous similar message [ 204.134258] Lustre: lustre-MDT0001: new disk, initializing [ 204.200781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.236786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.275492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.429982] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.915742] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.183579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.503631] Lustre: lustre-OST0000: new disk, initializing [ 225.507520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.515138] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.586637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.336041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.345053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.430250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.527702] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.382155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.503324] Lustre: lustre-OST0001: new disk, initializing [ 246.510761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 246.516949] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.574748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.487875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 252.512480] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.578782] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 253.197133] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.158248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.347666] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.591499] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.235927] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing yml_node [ 298.043651] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 301.422986] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 304.018811] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 305.351894] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 19:51:13 EDT 2026 [ 324.430828] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 326.048263] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 342.502045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 342.514265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.538613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.056347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.056601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.516699] Lustre: server umount lustre-MDT0000 complete [ 352.641541] LustreError: 10332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652321 with bad export cookie 5746665316794608427 [ 352.644557] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 352.660184] LustreError: 10332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 355.305795] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.310542] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.313903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.313912] Lustre: Skipped 3 previous similar messages [ 355.319573] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 355.386282] Lustre: Skipped 3 previous similar messages [ 357.353973] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.354171] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.363452] Lustre: Skipped 1 previous similar message [ 357.416566] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 359.566426] Lustre: server umount lustre-MDT0001 complete [ 364.785448] Lustre: server umount lustre-OST0000 complete [ 370.830268] Lustre: server umount lustre-OST0001 complete [ 378.492419] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 389.344940] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 427.390245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.624890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.665174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 427.778265] Lustre: lustre-MDT0000: new disk, initializing [ 427.866254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.878297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.591562] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 443.254564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.367248] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 443.410243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 443.412994] Lustre: Skipped 1 previous similar message [ 443.478919] Lustre: lustre-MDT0001: new disk, initializing [ 443.560629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.599770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 443.611436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.698512] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 458.039702] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.808192] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.860790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 470.202433] Lustre: lustre-OST0000: new disk, initializing [ 470.206670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 470.215636] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 470.394788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.426165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 471.454493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 471.624975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 477.686966] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 487.588995] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 492.307161] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 492.430212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 497.069285] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 497.384914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.245094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.247485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.278355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.291868] Lustre: Skipped 1 previous similar message [ 505.788069] Lustre: server umount lustre-OST0000 complete [ 510.432240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.441739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.458169] Lustre: Skipped 1 previous similar message [ 510.470189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.478279] Lustre: Skipped 1 previous similar message [ 516.434241] Lustre: server umount lustre-MDT0000 complete [ 517.614608] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.632513] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 519.511159] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652488 with bad export cookie 5746665316794609960 [ 519.519229] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.519817] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 519.746277] Lustre: server umount lustre-MDT0001 complete [ 523.670560] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 19:54:51 (1788652491) [ 529.742260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.169967] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.310502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.542921] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.901481] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 545.390443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.620301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 555.635105] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 560.685836] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.040058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 573.191102] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 580.951857] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 584.513071] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 591.349100] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.354441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 591.383714] Lustre: Skipped 3 previous similar messages [ 591.419213] Lustre: Skipped 4 previous similar messages [ 595.021142] Lustre: server umount lustre-MDT0000 complete [ 596.459809] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.488533] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 598.495085] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652567 with bad export cookie 5746665316794610835 [ 598.508307] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.521416] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 598.863387] Lustre: server umount lustre-MDT0001 complete [ 613.056504] Lustre: server umount lustre-OST0000 complete [ 633.934834] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 645.449651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.744916] LustreError: 23686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.778646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.785212] Lustre: Skipped 2 previous similar messages [ 648.555469] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 655.590402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.451293] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 661.918332] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 667.582449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.371083] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 682.875281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.071244] Lustre: lustre-OST0001: new disk, initializing [ 683.076149] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 683.090341] Lustre: 26000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 684.402361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 684.424686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 684.695687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 689.639867] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 698.952307] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 702.348487] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 715.237306] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.245668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.263958] Lustre: Skipped 2 previous similar messages [ 715.280635] Lustre: Skipped 1 previous similar message [ 721.218551] Lustre: server umount lustre-MDT0000 complete [ 725.120094] LustreError: 26971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652694 with bad export cookie 5746665316794611689 [ 725.122307] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.127328] LustreError: 26971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 725.560534] Lustre: server umount lustre-MDT0001 complete [ 739.793849] Lustre: server umount lustre-OST0000 complete [ 740.833747] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788652694/real 1788652694] req@ffff9424c8d98380 x1875537640535040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788652710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 743.382985] Lustre: server umount lustre-OST0001 complete [ 759.299815] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 768.471665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.724497] LustreError: 29023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.747886] LustreError: 29023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 772.354445] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 781.014519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.251836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 781.256090] Lustre: Skipped 4 previous similar messages [ 785.720930] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 788.763656] Lustre: 30172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 796.496772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.006408] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 813.027165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.402604] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 826.239192] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 829.937446] Lustre: 32049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 837.601994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 837.624630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.655845] Lustre: Skipped 2 previous similar messages [ 837.677712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.694031] Lustre: Skipped 6 previous similar messages [ 843.365383] Lustre: server umount lustre-MDT0000 complete [ 845.291755] LustreError: 30551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.304506] LustreError: 30551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 847.094933] LustreError: 29005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652816 with bad export cookie 5746665316794612977 [ 847.107689] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.116249] LustreError: 29005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 847.545897] Lustre: server umount lustre-MDT0001 complete [ 861.826644] Lustre: server umount lustre-OST0000 complete [ 875.063890] Lustre: server umount lustre-OST0001 complete [ 882.643881] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 20:00:50 (1788652850) [ 907.877086] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 918.190730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.207849] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 932.789223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.864559] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 940.844148] Lustre: 36019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 946.979933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.231199] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 962.525956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 969.501890] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 977.750067] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 981.038173] Lustre: 37898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 989.669089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 989.677862] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.692934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.706582] Lustre: Skipped 4 previous similar messages [ 989.716328] Lustre: Skipped 4 previous similar messages [ 991.894111] Lustre: server umount lustre-MDT0000 complete [ 994.786104] LustreError: 35607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.800111] LustreError: 35607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 995.033434] LustreError: 34853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652964 with bad export cookie 5746665316794614090 [ 995.045153] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.283635] Lustre: server umount lustre-MDT0001 complete [ 1008.308308] Lustre: server umount lustre-OST0000 complete [ 1022.035680] Lustre: server umount lustre-OST0001 complete [ 1030.148897] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 20:03:18 (1788652998) [ 1045.302543] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1055.471235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.961089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.965778] Lustre: Skipped 6 previous similar messages [ 1060.850492] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1069.622877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.101317] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1077.454948] Lustre: 41189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1082.616882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.359161] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1096.770925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.866672] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1109.127402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1112.187459] Lustre: 43066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1113.132644] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1113.135402] Lustre: Skipped 1 previous similar message [ 1116.132579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1121.910392] Lustre: server umount lustre-MDT0000 complete [ 1125.095883] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653094 with bad export cookie 5746665316794615196 [ 1125.107715] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1125.114748] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.880824] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.892627] Lustre: Skipped 7 previous similar messages [ 1131.464208] Lustre: server umount lustre-MDT0001 complete [ 1138.828848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.436288] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4fc041a89299ca03 [ 1150.442153] Lustre: MGC192.168.201.157@tcp: Connection restored to 0@lo (at 0@lo) [ 1156.344675] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1161.192351] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1164.624665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.632962] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1170.425799] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1170.433719] Lustre: Skipped 1 previous similar message [ 1179.903420] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1182.035490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1188.352183] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1190.738141] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1200.615207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1203.084161] Lustre: server umount lustre-MDT0000 complete [ 1207.385979] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653176 with bad export cookie 5746665316794616323 [ 1207.423520] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1207.433068] LustreError: 40022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1207.764563] Lustre: server umount lustre-MDT0001 complete [ 1223.152410] Lustre: server umount lustre-OST0000 complete [ 1237.845626] Lustre: server umount lustre-OST0001 complete [ 1246.799981] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 20:06:54 (1788653214) [ 1260.442724] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1272.644267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.162123] LustreError: 47675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.180530] LustreError: 47675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [ 1279.243112] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1288.439231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.341630] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1297.488628] Lustre: 48828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1304.824209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.015094] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1323.218620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.735689] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1338.104753] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1342.973785] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1359.334899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.349785] Lustre: Skipped 22 previous similar messages [ 1364.405864] Lustre: server umount lustre-MDT0000 complete [ 1368.341740] LustreError: 50716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653337 with bad export cookie 5746665316794617114 [ 1368.363471] LustreError: 50716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1368.363976] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1368.874277] Lustre: server umount lustre-MDT0001 complete [ 1384.286668] Lustre: server umount lustre-OST0000 complete [ 1385.183126] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788653338/real 1788653338] req@ffff9425f6c29880 x1875537640779392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788653354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1385.231851] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1385.249841] Lustre: Skipped 8 previous similar messages [ 1388.384818] Lustre: server umount lustre-OST0001 complete [ 1398.664337] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 20:09:26 (1788653366) [ 1401.172528] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1403.673697] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 20:09:30 (1788653370) [ 1416.335541] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 1426.615027] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1452.364533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.754141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1452.852322] Lustre: lustre-MDT0000: new disk, initializing [ 1452.941946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1458.132746] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1469.128240] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1476.543722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1476.788658] Lustre: lustre-OST0000: new disk, initializing [ 1476.792908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1476.797767] Lustre: Skipped 1 previous similar message [ 1476.816744] Lustre: 54909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1478.566390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1478.573585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1478.642431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1483.467513] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1493.402264] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1495.597818] Lustre: DEBUG MARKER: create test files [ 1529.830799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1535.896768] Lustre: server umount lustre-OST0000 complete [ 1539.994246] Lustre: server umount lustre-MDT0000 complete [ 1552.320949] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1555.867841] Key type lgssc unregistered [ 1556.247934] LNet: 56538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.256950] LNetError: 56538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.287368] LNet: Removed LNI 192.168.201.157@tcp [ 1558.332200] Key type .llcrypt unregistered [ 1558.333793] Key type ._llcrypt unregistered [ 1585.524983] Key type ._llcrypt registered [ 1585.527374] Key type .llcrypt registered [ 1585.584732] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1586.510667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.523474] alg: No test for adler32 (adler32-zlib) [ 1587.494577] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1587.755837] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1589.391256] Key type lgssc registered [ 1590.734820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1598.823691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1598.845590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.664689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.271594] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1617.558965] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1624.988944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.352939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.368274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:101 to 0x240000400:129) [ 1626.466778] Lustre: 59252:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1626.474834] Lustre: 59252:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1626.489970] Lustre: 59252:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1626.504422] Lustre: 59252:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1626.512245] Lustre: 59252:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1626.520948] Lustre: 59252:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1632.476951] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1641.094115] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.840861] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1653.791961] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1662.405571] Lustre: server umount lustre-OST0000 complete [ 1677.070957] Lustre: server umount lustre-MDT0000 complete [ 1688.313916] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1691.067390] Key type lgssc unregistered [ 1691.455965] LNet: 60863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.479052] LNetError: 60863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.519202] LNet: Removed LNI 192.168.201.157@tcp [ 1693.583444] Key type .llcrypt unregistered [ 1693.586958] Key type ._llcrypt unregistered [ 1725.387035] Key type ._llcrypt registered [ 1725.390264] Key type .llcrypt registered [ 1725.521857] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1726.757707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1726.805867] alg: No test for adler32 (adler32-zlib) [ 1728.067068] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1728.389950] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1730.127263] Key type lgssc registered [ 1731.723382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.884766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1741.923793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.569510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.582405] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1758.687833] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1767.212446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1767.604776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.637993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1768.729866] Lustre: 63707:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1768.744325] Lustre: 63707:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1768.753654] Lustre: 63707:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1768.762258] Lustre: 63707:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1768.774760] Lustre: 63707:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1768.786793] Lustre: 63707:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1774.822157] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1782.303389] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1787.572797] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1799.649535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.674072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.316688] Lustre: server umount lustre-OST0000 complete [ 1807.033672] Lustre: server umount lustre-MDT0000 complete [ 1819.602711] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1822.331951] Key type lgssc unregistered [ 1822.576949] LNet: 65264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.584842] LNetError: 65264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.602535] LNet: Removed LNI 192.168.201.157@tcp [ 1823.682591] Key type .llcrypt unregistered [ 1823.687158] Key type ._llcrypt unregistered [ 1841.968911] Key type ._llcrypt registered [ 1841.970906] Key type .llcrypt registered [ 1842.025604] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 1858.423961] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1859.760852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.923890] alg: No test for adler32 (adler32-zlib) [ 1861.093966] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1861.361388] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1863.096415] Key type lgssc registered [ 1864.544796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1900.479766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1900.514450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.892994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1901.944707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1902.023708] Lustre: lustre-MDT0000: new disk, initializing [ 1902.109274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.132755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1906.772470] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1918.257752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.395910] Lustre: 68415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1918.443085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1918.446239] Lustre: Skipped 1 previous similar message [ 1918.511534] Lustre: lustre-MDT0001: new disk, initializing [ 1918.603225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.632941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1918.656382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1924.029413] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1933.888362] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1940.223590] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1947.497741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.731939] Lustre: lustre-OST0000: new disk, initializing [ 1947.735198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1947.740581] Lustre: 70381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1947.865917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.362187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1953.375837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1953.521357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1953.973219] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1964.522536] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.705685] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1970.061342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1975.631880] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1975.895956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1979.366920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.371963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.383773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.389346] LustreError: Skipped 1 previous similar message [ 1979.401501] Lustre: Skipped 1 previous similar message [ 1984.373794] Lustre: server umount lustre-OST0000 complete [ 1989.602213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.608946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.616873] Lustre: Skipped 2 previous similar messages [ 1989.628406] Lustre: Skipped 1 previous similar message [ 1994.832366] Lustre: server umount lustre-MDT0000 complete [ 1999.053174] LustreError: 68407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653968 with bad export cookie 17256817643826290286 [ 1999.057820] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.058840] LustreError: 68407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1999.601637] Lustre: server umount lustre-MDT0001 complete [ 2008.827304] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 20:19:36 (1788653976) [ 2021.374279] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2059.993676] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 2068.784909] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2107.587041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.970444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2108.055554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2108.156518] Lustre: lustre-MDT0000: new disk, initializing [ 2108.220315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.237916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2112.188281] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2123.216398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.411515] Lustre: 76405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2123.545518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2123.548134] Lustre: Skipped 1 previous similar message [ 2123.610853] Lustre: lustre-MDT0001: new disk, initializing [ 2123.670342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.707437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2123.716601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2129.393704] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2140.905535] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2147.254302] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2154.926533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.370499] Lustre: lustre-OST0000: new disk, initializing [ 2155.373847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2155.379185] Lustre: 78374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2155.512969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.282628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2162.294652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2162.413932] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2162.425670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2170.938525] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.150790] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2176.292903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2180.204562] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2180.330086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2183.136763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.139841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.148568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.154498] Lustre: Skipped 1 previous similar message [ 2187.949613] Lustre: server umount lustre-OST0000 complete [ 2193.380048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.385881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.394549] Lustre: Skipped 2 previous similar messages [ 2193.408502] Lustre: Skipped 1 previous similar message [ 2198.499495] LustreError: 76417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.500784] Lustre: server umount lustre-MDT0000 complete [ 2198.513762] LustreError: 76417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2203.033320] LustreError: 77200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654172 with bad export cookie 17256817643826291168 [ 2203.036918] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.041250] LustreError: 77200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2203.600116] Lustre: server umount lustre-MDT0001 complete [ 2213.378483] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:23:00 (1788654180) [ 2220.096813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.479912] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.589104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.639332] LustreError: 80786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.039747] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2230.752906] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.461141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.345603] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2248.636445] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2254.049132] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.102106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.621678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.624792] Lustre: Skipped 1 previous similar message [ 2269.072789] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2276.669733] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2279.504350] Lustre: 83460:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 2d757e39-6e1e-4c6c-9aef-91e9ee5edb24 at adminstrative request [ 2280.048891] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.069324] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2290.663812] Lustre: MGS: Client 2d757e39-6e1e-4c6c-9aef-91e9ee5edb24 (at 0@lo) reconnecting [ 2290.672487] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xef7c88376df5a552 to 0xef7c88376df5a7f2 [ 2290.688495] Lustre: MGC192.168.201.157@tcp: Connection restored to 0@lo (at 0@lo) [ 2316.439462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2316.606403] Lustre: lustre-OST0001: new disk, initializing [ 2316.608724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2316.614286] Lustre: 83854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2321.985406] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2325.629625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2325.636181] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2325.693412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2331.491325] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2336.224924] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2336.231985] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.240281] LustreError: Skipped 2 previous similar messages [ 2336.242073] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2336.256687] Lustre: Skipped 1 previous similar message [ 2336.268731] Lustre: Skipped 1 previous similar message [ 2340.643601] Lustre: server umount lustre-OST0001 complete [ 2341.348812] LustreError: 82609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.360884] LustreError: 82609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2346.474508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.489684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.495320] Lustre: Skipped 1 previous similar message [ 2350.510406] Lustre: server umount lustre-OST0000 complete [ 2360.478983] Lustre: server umount lustre-MDT0000 complete [ 2361.825317] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.855910] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2364.252599] LustreError: 80766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654333 with bad export cookie 17256817643826292722 [ 2364.263926] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2364.269445] LustreError: 80766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2364.541758] Lustre: server umount lustre-MDT0001 complete [ 2372.371653] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:25:40 (1788654340) [ 2373.709957] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2375.462919] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:25:43 (1788654343) [ 2377.138628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2379.252917] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:25:46 (1788654346) [ 2380.493932] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2382.255407] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 20:25:49 (1788654349) [ 2383.823725] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2385.725900] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:25:53 (1788654353) [ 2387.707456] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2394.944663] loop: module loaded [ 2394.956853] loop0: detected capacity change from 0 to 4294967296 [ 2395.044491] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2396.772258] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2400.628392] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2405.692497] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2433.359207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2433.426324] LustreError: 87977:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2433.434353] LustreError: 87977:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2433.439470] LustreError: 87977:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2433.451324] LustreError: 87977:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2452.787301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 2464.239831] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2499.223647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.545913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2499.617903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2499.821257] Lustre: lustre-MDT0000: new disk, initializing [ 2499.942304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.953289] Lustre: Skipped 1 previous similar message [ 2499.993117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2505.521737] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2517.549583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.712392] Lustre: 90843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2517.935351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2522.736789] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2531.859462] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2538.278835] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2545.191937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2545.526554] Lustre: 92810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2553.075474] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2553.442021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2553.544084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2563.478402] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2568.665604] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2568.883312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2573.888834] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2574.193737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2579.426693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.433867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2579.434440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.434444] Lustre: Skipped 3 previous similar messages [ 2579.444020] Lustre: Skipped 3 previous similar messages [ 2582.313532] Lustre: server umount lustre-OST0000 complete [ 2589.699429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2593.073900] Lustre: server umount lustre-MDT0000 complete [ 2594.784255] LustreError: 90847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.800842] LustreError: 90847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2598.412074] LustreError: 92809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654567 with bad export cookie 17256817643826293331 [ 2598.419357] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.426194] LustreError: 92809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2598.923462] Lustre: server umount lustre-MDT0001 complete [ 2609.612729] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:29:36 (1788654576) [ 2616.777594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.644834] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2627.553034] LustreError: 95227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.585284] LustreError: 95227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2632.697927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.138669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2633.143904] Lustre: Skipped 3 previous similar messages [ 2637.326189] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2644.464736] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2650.474775] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2657.309140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2665.131650] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2671.943732] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.739386] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2677.008197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2681.960605] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2682.175424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2689.239361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.518666] Lustre: lustre-OST0001: new disk, initializing [ 2689.523024] Lustre: Skipped 2 previous similar messages [ 2689.535401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2689.541722] Lustre: Skipped 3 previous similar messages [ 2689.553640] Lustre: 98595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2697.200060] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2699.373440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2699.377518] Lustre: Skipped 2 previous similar messages [ 2699.383958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2699.476649] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2707.199520] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2712.364707] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2712.582937] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2717.711848] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2717.986615] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2738.541043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2763.348721] Lustre: Failing over lustre-MDT0000 [ 2763.933531] Lustre: server umount lustre-MDT0000 complete [ 2766.337971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.363055] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.374747] LustreError: 95227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.394407] Lustre: Skipped 3 previous similar messages [ 2766.454109] LustreError: 95227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2782.687329] Lustre: 66635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788654735/real 1788654735] req@0000000036f6dd11 x1875539478715776/t0(0) o400->MGC192.168.201.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788654751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2782.735451] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.344983] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2790.347425] LDISKFS-fs (dm-0): recovery complete [ 2790.352992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.910526] LustreError: 66634:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e54fc10f x1875539478723584/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2792.326316] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2796.964912] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2796.973592] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2796.973713] Lustre: 95222:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2797.567731] Lustre: 95226:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2797.577971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2797.588130] Lustre: 95226:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2797.693281] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2798.023068] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2798.353694] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2798.377414] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2798.698744] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2798.711144] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2798.722321] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2798.731196] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2799.382186] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2799.405473] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2799.727406] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2799.739468] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2801.033579] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2801.042553] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2801.696749] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2801.712328] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2802.039488] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2802.048428] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2805.232516] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2805.246035] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2805.889893] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2805.901754] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2806.223493] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2806.229058] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2813.528286] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2813.549462] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 25 previous similar messages [ 2814.202191] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2814.218839] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2814.544201] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2814.567052] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2829.779146] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2829.787981] Lustre: 101450:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2830.423839] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2830.436391] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2830.754518] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2830.759876] LustreError: 101450:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2856.231507] Lustre: 101450:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2856.235541] Lustre: 101450:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2856.241374] Lustre: 101450:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a85c5380-cd17-478b-8994-713ff774ae94@192.168.201.57@tcp [ 2856.251703] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2856.255569] Lustre: 101450:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@000000001b35fb83 x1875539461740800/t0(8589934780) o35->a85c5380-cd17-478b-8994-713ff774ae94@192.168.201.57@tcp:191/0 lens 392/0 e 0 to 0 dl 1788654836 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2856.294285] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2856.317274] Lustre: lustre-MDT0000: Denying connection for new client a85c5380-cd17-478b-8994-713ff774ae94 (at 192.168.201.57@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2856.327641] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2856.382936] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2856.400720] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2856.404899] Lustre: Skipped 2 previous similar messages [ 2856.449821] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2856.460649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2872.801653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2872.813444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.837414] Lustre: Skipped 2 previous similar messages [ 2872.843173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.850097] Lustre: Skipped 3 previous similar messages [ 2878.319753] Lustre: server umount lustre-OST0000 complete [ 2894.818120] LustreError: 97048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.837107] LustreError: 97048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 2896.351312] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2896.521784] Lustre: server umount lustre-OST0001 complete [ 2907.217572] Lustre: server umount lustre-MDT0000 complete [ 2911.407561] LustreError: 95207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654880 with bad export cookie 17256817643826344557 [ 2911.421638] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2911.831783] Lustre: server umount lustre-MDT0001 complete [ 2920.901638] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:34:48 (1788654888) [ 2928.763842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2928.769334] Lustre: *** cfs_fail_loc=197, val=0*** [ 2944.288949] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2949.408653] LustreError: 103427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.157@tcp: failed processing log, type 4: rc = -110 [ 2954.641810] Lustre: *** cfs_fail_loc=197, val=0*** [ 2954.648036] Lustre: Skipped 65 previous similar messages [ 2975.199429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.205717] Lustre: Skipped 3 previous similar messages [ 2982.706834] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2991.570501] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2994.365558] Lustre: server umount lustre-OST0000 complete [ 3003.224102] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:36:10 (1788654970) [ 3026.224515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.448842] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3042.962406] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3046.454149] LustreError: 105082:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3046.464714] LustreError: 105082:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3046.470713] LustreError: 105082:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3048.933619] Lustre: server umount lustre-MDT0000 complete [ 3084.747826] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 3095.767816] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3134.535948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.836962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3134.858585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3134.912232] Lustre: lustre-MDT0000: new disk, initializing [ 3135.018941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3140.398100] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3151.876895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.021352] Lustre: 109257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3152.285669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3156.639430] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3166.819526] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3172.807422] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3180.281757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.560764] Lustre: 111224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3186.422049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3187.442271] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3196.747743] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3201.687554] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3201.883536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3206.572977] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3206.794207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3212.258329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3212.269944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.274744] LustreError: Skipped 5 previous similar messages [ 3212.300954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3212.308932] Lustre: Skipped 6 previous similar messages [ 3212.328290] Lustre: Skipped 14 previous similar messages [ 3215.608371] Lustre: server umount lustre-OST0000 complete [ 3226.512248] Lustre: server umount lustre-MDT0000 complete [ 3227.631625] LustreError: 109264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.652631] LustreError: 109264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3230.460436] LustreError: 109249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655199 with bad export cookie 17256817643826355253 [ 3230.462569] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.467411] LustreError: 109249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3230.869590] Lustre: server umount lustre-MDT0001 complete [ 3240.729994] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:40:08 (1788655208) [ 3251.673514] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3254.086892] Key type lgssc unregistered [ 3254.257874] LNet: 113712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.262822] LNetError: 113712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.273639] LNet: Removed LNI 192.168.201.157@tcp [ 3255.146322] Key type .llcrypt unregistered [ 3255.147988] Key type ._llcrypt unregistered [ 3276.815110] Key type ._llcrypt registered [ 3276.817503] Key type .llcrypt registered [ 3276.985213] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 3293.938711] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3295.072845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3295.174195] alg: No test for adler32 (adler32-zlib) [ 3296.448321] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 3296.773473] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3298.455182] Key type lgssc registered [ 3299.952900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3343.438524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3343.477119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.759753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3344.780884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3344.845995] Lustre: lustre-MDT0000: new disk, initializing [ 3344.947559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.962551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3349.492155] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3360.693165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.189645] Lustre: lustre-OST0000: new disk, initializing [ 3361.199507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3361.203727] Lustre: Skipped 1 previous similar message [ 3361.211284] Lustre: 117886:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3361.310785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.054117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3368.064968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3368.086820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3369.936372] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3381.018499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3381.299407] Lustre: lustre-OST0001: new disk, initializing [ 3381.314576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3381.339634] Lustre: 118748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3381.473505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.216991] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3389.713842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3389.747964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3389.821533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3400.163852] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.167154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.174815] Lustre: Skipped 1 previous similar message [ 3400.183168] Lustre: Skipped 1 previous similar message [ 3404.345759] Lustre: server umount lustre-MDT0000 complete [ 3411.318713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3416.413096] LustreError: 117074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655385 with bad export cookie 829926893015000375 [ 3416.425703] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.427480] LustreError: 117074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3416.563457] Lustre: server umount lustre-OST0000 complete [ 3420.836381] Lustre: server umount lustre-OST0001 complete [ 3430.272602] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 20:43:17 (1788655397) [ 3451.679545] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 20:43:39 (1788655419) [ 3462.377607] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3485.452781] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 20:44:12 (1788655452) [ 3525.168287] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 3535.361482] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3574.498958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.772675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3574.808274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3574.973379] Lustre: lustre-MDT0000: new disk, initializing [ 3575.210597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.234173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3580.823940] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3592.087967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.218732] Lustre: 126499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3592.270265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3592.274114] Lustre: Skipped 1 previous similar message [ 3592.345350] Lustre: lustre-MDT0001: new disk, initializing [ 3592.422188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3592.461556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3592.466529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3597.684448] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3607.890177] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3613.977569] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3621.442507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.738157] Lustre: lustre-OST0000: new disk, initializing [ 3621.740788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3621.747487] Lustre: 128471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3621.818908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.969470] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3630.168409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3630.179606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3630.275343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3639.564162] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3645.371995] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3645.629161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3650.758054] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3650.906982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3656.163282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.180208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3656.198723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.243962] Lustre: Skipped 1 previous similar message [ 3659.536215] Lustre: server umount lustre-OST0000 complete [ 3664.356972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3664.364823] LustreError: Skipped 1 previous similar message [ 3664.366908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.376661] Lustre: Skipped 1 previous similar message [ 3664.381814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.406080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.408244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.502678] Lustre: server umount lustre-MDT0000 complete [ 3675.292938] LustreError: 127295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655644 with bad export cookie 829926893015001593 [ 3675.310708] LustreError: 127295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3675.322713] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3675.955987] Lustre: server umount lustre-MDT0001 complete [ 3687.794515] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 20:47:35 (1788655655) [ 3695.971054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.668731] LustreError: 130886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3696.691652] LustreError: 130886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3696.856588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3701.986472] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3702.249229] LustreError: 130887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3706.339749] LustreError: 130886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3711.143739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.476492] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3727.282718] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3734.454678] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3742.601783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.099537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.102410] Lustre: Skipped 1 previous similar message [ 3750.925112] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3759.704957] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.445493] Lustre: *** cfs_fail_loc=159, val=0*** [ 3775.447673] Lustre: Skipped 1 previous similar message [ 3791.844261] Lustre: lustre-MDT0000: Client e181ac4d-a8c6-4fe9-9906-73ef81365315 (at 192.168.201.57@tcp) reconnecting [ 3793.565569] Lustre: *** cfs_fail_loc=159, val=0*** [ 3793.573250] Lustre: Skipped 4 previous similar messages [ 3809.267646] Lustre: lustre-MDT0000: Client e181ac4d-a8c6-4fe9-9906-73ef81365315 (at 192.168.201.57@tcp) reconnecting [ 3828.192277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3828.196418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.199907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.202308] Lustre: Skipped 1 previous similar message [ 3828.205113] Lustre: Skipped 1 previous similar message [ 3833.488516] Lustre: server umount lustre-OST0000 complete [ 3839.460900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3839.472275] LustreError: Skipped 1 previous similar message [ 3839.475431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.485568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.488017] Lustre: Skipped 2 previous similar messages [ 3843.428837] Lustre: server umount lustre-MDT0000 complete [ 3846.635414] LustreError: 133566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.666855] LustreError: 133566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3849.052462] LustreError: 133632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655818 with bad export cookie 829926893015002482 [ 3849.058341] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.085308] LustreError: 133632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3849.768831] Lustre: server umount lustre-MDT0001 complete [ 3862.125301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3864.611169] Key type lgssc unregistered [ 3864.911614] LNet: 134687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.922417] LNetError: 134687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3864.948401] LNet: Removed LNI 192.168.201.157@tcp [ 3865.927184] Key type .llcrypt unregistered [ 3865.929088] Key type ._llcrypt unregistered [ 3890.505233] Key type ._llcrypt registered [ 3890.508490] Key type .llcrypt registered [ 3890.655534] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3891.825494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3891.843849] alg: No test for adler32 (adler32-zlib) [ 3893.047953] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 3893.347672] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3894.975240] Key type lgssc registered [ 3896.604348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3904.698180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3904.742909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.263521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.847821] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3915.881404] LustreError: 136100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.916660] LustreError: 136100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3921.399479] LustreError: 136099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.765740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.092198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3930.333308] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3939.949270] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3947.576733] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3957.667196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3958.497185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3968.798551] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3975.997143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3977.769392] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3987.009661] Lustre: *** cfs_fail_loc=159, val=0*** [ 4003.291429] Lustre: lustre-MDT0000: Client 16ff404e-57fa-459b-a310-8b8ca0ac3957 (at 192.168.201.57@tcp) reconnecting [ 4005.712938] Lustre: *** cfs_fail_loc=159, val=0*** [ 4021.249047] Lustre: lustre-MDT0001: Client 16ff404e-57fa-459b-a310-8b8ca0ac3957 (at 192.168.201.57@tcp) reconnecting [ 4022.541372] Lustre: *** cfs_fail_loc=159, val=0*** [ 4022.546411] Lustre: Skipped 3 previous similar messages [ 4038.109500] Lustre: lustre-MDT0001: Client 16ff404e-57fa-459b-a310-8b8ca0ac3957 (at 192.168.201.57@tcp) reconnecting [ 4042.169947] Lustre: 139135:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4044.335642] Lustre: *** cfs_fail_loc=159, val=0*** [ 4044.342537] Lustre: Skipped 4 previous similar messages [ 4060.149658] Lustre: lustre-MDT0000: Client 6177b76f-b141-4be1-9cb1-2a7c4b243e29 (at 192.168.201.57@tcp) reconnecting [ 4062.088919] Lustre: *** cfs_fail_loc=159, val=0*** [ 4062.096270] Lustre: Skipped 15 previous similar messages [ 4078.583523] Lustre: lustre-MDT0000: Client 6177b76f-b141-4be1-9cb1-2a7c4b243e29 (at 192.168.201.57@tcp) reconnecting [ 4080.518103] Lustre: 139426:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4099.051605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4099.052482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.085704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.132209] Lustre: Skipped 1 previous similar message [ 4104.166220] LustreError: 139441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.190139] LustreError: 139441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4104.500107] Lustre: server umount lustre-OST0000 complete [ 4109.284596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.290512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.305102] Lustre: Skipped 1 previous similar message [ 4109.323256] Lustre: Skipped 2 previous similar messages [ 4114.401736] LustreError: 139196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.475779] Lustre: server umount lustre-MDT0000 complete [ 4118.385503] LustreError: 139256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656087 with bad export cookie 9870110939369726523 [ 4118.392809] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4118.394932] LustreError: 139256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4118.778903] Lustre: server umount lustre-MDT0001 complete [ 4129.734210] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4132.190825] Key type lgssc unregistered [ 4132.468718] LNet: 140505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4132.474283] LNetError: 140505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4133.544991] LNet: Removed LNI 192.168.201.157@tcp [ 4134.460613] Key type .llcrypt unregistered [ 4134.465445] Key type ._llcrypt unregistered [ 4157.964814] Key type ._llcrypt registered [ 4157.970577] Key type .llcrypt registered [ 4158.123700] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4158.944670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4158.984080] alg: No test for adler32 (adler32-zlib) [ 4160.181473] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 4160.485375] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4162.159198] Key type lgssc registered [ 4163.794262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4172.112767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4172.137510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.643219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.904278] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4183.014233] LustreError: 141918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.038168] LustreError: 141918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4188.139182] LustreError: 141917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4189.396383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.705868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4194.498181] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4202.123922] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4207.986561] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4215.385720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4215.813969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.859302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4222.873313] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4232.023991] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4234.238338] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4242.077709] Lustre: 144498:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4244.894597] Lustre: 144546:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4246.348624] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4247.185165] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4250.797389] Lustre: 144693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4251.696482] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4252.736679] Lustre: 144791:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4258.792774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4258.801638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.812249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.816285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4259.817492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.839257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.864849] Lustre: Skipped 1 previous similar message [ 4262.697713] Lustre: server umount lustre-OST0000 complete [ 4266.466300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.473335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.485256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.048837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.053748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.172616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.179396] Lustre: Skipped 2 previous similar messages [ 4280.287186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4280.656327] Lustre: server umount lustre-MDT0000 complete [ 4285.065937] LustreError: 144655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656254 with bad export cookie 12443731632637422983 [ 4285.074584] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4285.084897] LustreError: 144655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4285.551427] Lustre: server umount lustre-MDT0001 complete [ 4299.736725] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4303.069779] Key type lgssc unregistered [ 4303.337699] LNet: 145853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4303.348907] LNetError: 145853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4304.421191] LNet: Removed LNI 192.168.201.157@tcp [ 4305.255669] Key type .llcrypt unregistered [ 4305.257505] Key type ._llcrypt unregistered [ 4331.785657] Key type ._llcrypt registered [ 4331.787494] Key type .llcrypt registered [ 4331.934204] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4333.727448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4333.788498] alg: No test for adler32 (adler32-zlib) [ 4334.972391] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 4335.178812] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4336.823782] Key type lgssc registered [ 4338.049587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4347.008111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4347.036963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.803664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.032777] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4358.118945] LustreError: 147263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.138202] LustreError: 147263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4363.240724] LustreError: 147264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.218095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.762758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4370.265878] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4378.824499] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4385.865186] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4393.933944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4394.352224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.317334] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4409.721984] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4410.948807] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4417.904813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4422.586144] Lustre: *** cfs_fail_loc=159, val=0*** [ 4422.598031] Lustre: Skipped 3 previous similar messages [ 4438.011502] Lustre: lustre-MDT0000: Client f2d07b38-74bc-4c67-84bb-69b0fdd88d7e (at 192.168.201.57@tcp) reconnecting [ 4443.616959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4443.621769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.627528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4443.631358] Lustre: Skipped 1 previous similar message [ 4443.651384] Lustre: Skipped 1 previous similar message [ 4445.333413] Lustre: server umount lustre-OST0000 complete [ 4451.810166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4451.825891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.851464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.857031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.870971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.981488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.993603] Lustre: Skipped 2 previous similar messages [ 4463.074070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.080077] Lustre: Skipped 1 previous similar message [ 4464.095159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4464.291461] Lustre: server umount lustre-MDT0000 complete [ 4468.201206] LustreError: 147264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.221909] LustreError: 147264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4468.570183] LustreError: 147245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656437 with bad export cookie 3733904652244925963 [ 4468.582876] LustreError: 147245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4468.583367] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4468.923749] Lustre: server umount lustre-MDT0001 complete [ 4480.756249] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4485.060917] Key type lgssc unregistered [ 4485.439747] LNet: 150947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4485.451575] LNetError: 150947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4486.505862] LNet: Removed LNI 192.168.201.157@tcp [ 4487.575479] Key type .llcrypt unregistered [ 4487.578225] Key type ._llcrypt unregistered [ 4516.151778] Key type ._llcrypt registered [ 4516.156635] Key type .llcrypt registered [ 4516.261634] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4517.577251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4517.588309] alg: No test for adler32 (adler32-zlib) [ 4518.829535] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 4519.004951] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4520.730877] Key type lgssc registered [ 4522.179914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4529.638922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4529.654705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.151600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.900417] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4541.410244] LustreError: 152370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.434891] LustreError: 152370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4545.753047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.201677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4551.844966] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4559.677418] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4566.279544] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4573.529348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4574.217438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.390799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4580.648269] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4588.373818] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.686826] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4613.874103] Lustre: lustre-MDT0001: evicting 69874202-d59c-47ad-8247-803c9f84b16b (at 192.168.201.57@tcp) by administrative request [ 4613.884235] Lustre: Skipped 1 previous similar message [ 4643.814882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.818692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.825256] LustreError: Skipped 1 previous similar message [ 4643.838424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4643.849680] Lustre: Skipped 1 previous similar message [ 4643.880163] Lustre: Skipped 1 previous similar message [ 4648.943891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.951070] Lustre: Skipped 1 previous similar message [ 4654.049551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4657.120416] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4657.271605] Lustre: server umount lustre-OST0000 complete [ 4663.777632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4663.787698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.813647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4663.830634] Lustre: Skipped 1 previous similar message [ 4667.571442] Lustre: server umount lustre-MDT0000 complete [ 4669.413508] LustreError: 155294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.438033] LustreError: 155294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4671.578880] LustreError: 152351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656640 with bad export cookie 5045697841728582015 [ 4671.582970] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4671.589055] LustreError: 152351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4672.049828] Lustre: server umount lustre-MDT0001 complete [ 4685.315701] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4688.131975] Key type lgssc unregistered [ 4688.482668] LNet: 156445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4688.488833] LNetError: 156445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4688.502207] LNet: Removed LNI 192.168.201.157@tcp [ 4689.388233] Key type .llcrypt unregistered [ 4689.390533] Key type ._llcrypt unregistered [ 4758.971968] Key type ._llcrypt registered [ 4758.975757] Key type .llcrypt registered [ 4759.087945] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4760.709750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4760.742100] alg: No test for adler32 (adler32-zlib) [ 4761.947759] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 4762.259376] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4763.975227] Key type lgssc registered [ 4765.224800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4774.332730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4774.355332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.926915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.746647] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4786.147674] LustreError: 158941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.185053] LustreError: 158941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4789.555082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.109882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4795.301978] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4802.681762] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4808.459139] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4815.108709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4815.420173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.445940] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4821.430598] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4821.552612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4828.488848] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4836.832692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.837813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.857110] Lustre: Skipped 1 previous similar message [ 4836.859177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.886728] Lustre: Skipped 1 previous similar message [ 4841.956674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4842.676735] Lustre: server umount lustre-OST0000 complete [ 4846.567553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4846.579467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.587217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4846.589708] Lustre: Skipped 1 previous similar message [ 4852.192845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4852.200386] Lustre: Skipped 3 previous similar messages [ 4857.312506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.384685] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.605255] Lustre: server umount lustre-MDT0000 complete [ 4862.432807] LustreError: 158937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.443445] LustreError: 158937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4864.226070] LustreError: 158921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656833 with bad export cookie 11263936982340574657 [ 4864.236809] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.494227] Lustre: server umount lustre-MDT0001 complete [ 4874.316500] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4877.229654] Key type lgssc unregistered [ 4877.687501] LNet: 162479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4877.695310] LNetError: 162479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4877.713609] LNet: Removed LNI 192.168.201.157@tcp [ 4878.553388] Key type .llcrypt unregistered [ 4878.555634] Key type ._llcrypt unregistered [ 4900.106538] Key type ._llcrypt registered [ 4900.109395] Key type .llcrypt registered [ 4900.245346] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4900.693480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4900.704412] alg: No test for adler32 (adler32-zlib) [ 4901.856729] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 4902.059871] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4903.759325] Key type lgssc registered [ 4905.380923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4920.687490] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 21:08:08 (1788656888) [ 4937.552195] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 4947.713115] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4983.384164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4983.398964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.615882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4984.649758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4984.729461] Lustre: lustre-MDT0000: new disk, initializing [ 4984.806836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4984.832103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4990.292933] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5001.066516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.285192] Lustre: 166738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5001.361988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5001.369796] Lustre: Skipped 1 previous similar message [ 5001.502211] Lustre: lustre-MDT0001: new disk, initializing [ 5001.621681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5001.663758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5001.680120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.939368] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5016.670075] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5021.961698] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5028.056040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5028.398475] Lustre: lustre-OST0000: new disk, initializing [ 5028.406746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5028.421050] Lustre: 168708:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5028.592071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.622501] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5034.624776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5034.632814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5034.775598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5043.204967] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5055.458368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5055.472835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.491667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5055.503849] Lustre: Skipped 1 previous similar message [ 5055.518086] Lustre: Skipped 1 previous similar message [ 5059.399284] Lustre: server umount lustre-OST0000 complete [ 5065.702816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.716826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5065.730682] Lustre: Skipped 1 previous similar message [ 5065.749407] Lustre: Skipped 1 previous similar message [ 5069.865259] Lustre: server umount lustre-MDT0000 complete [ 5073.967323] LustreError: 169574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657043 with bad export cookie 715558982700361106 [ 5073.980096] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.320780] Lustre: server umount lustre-MDT0001 complete [ 5085.310300] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 5087.456434] Key type lgssc unregistered [ 5087.697188] LNet: 170616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5087.704152] LNetError: 170616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5087.723792] LNet: Removed LNI 192.168.201.157@tcp [ 5088.367509] Key type .llcrypt unregistered [ 5088.369642] Key type ._llcrypt unregistered [ 5111.825200] Key type ._llcrypt registered [ 5111.830723] Key type .llcrypt registered [ 5111.973124] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5112.832816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5112.857554] alg: No test for adler32 (adler32-zlib) [ 5114.054490] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5114.329079] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 5116.071205] Key type lgssc registered [ 5117.115527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5123.631462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5123.660336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.075951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5129.161029] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5135.335269] LustreError: 172027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.341764] LustreError: 172027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5135.582378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.859444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5139.727324] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5146.916431] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5152.373362] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5158.901547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5159.329284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.811223] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5170.616664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5174.646456] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5181.467285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5181.476617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.487133] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5181.492505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.499508] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5182.523801] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5182.550774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.567172] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5182.579181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.592768] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5183.542646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5183.571450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.592171] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5183.598716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.611269] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5184.457256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.520794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.480213] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5186.484732] Lustre: Skipped 2 previous similar messages [ 5186.508274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.524102] Lustre: Skipped 2 previous similar messages [ 5186.532710] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5186.543907] Lustre: Skipped 2 previous similar messages [ 5186.548278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.558755] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5186.580182] Lustre: Skipped 2 previous similar messages [ 5187.368227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.479352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.423574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.352719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.192422] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5191.196425] Lustre: Skipped 4 previous similar messages [ 5191.208624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.223398] Lustre: Skipped 4 previous similar messages [ 5191.228192] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5191.239579] Lustre: Skipped 4 previous similar messages [ 5191.244383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.258699] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5191.270493] Lustre: Skipped 4 previous similar messages [ 5191.977038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.940956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.800049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.901239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.767315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.498383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.301415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.181964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.946851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.885422] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5199.889540] Lustre: Skipped 9 previous similar messages [ 5199.898906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.903714] Lustre: Skipped 9 previous similar messages [ 5199.906116] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5199.908606] Lustre: Skipped 9 previous similar messages [ 5199.911164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.916644] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5199.924035] Lustre: Skipped 9 previous similar messages [ 5200.631413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.417453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.208674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.082176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.014322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.821964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.603163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.495220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.474236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.399245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.201585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.886556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.828933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.883861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.681839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.559324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.651412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.662897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.708941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5216.713445] Lustre: Skipped 18 previous similar messages [ 5216.726481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.736316] Lustre: Skipped 18 previous similar messages [ 5216.742152] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5216.748262] Lustre: Skipped 18 previous similar messages [ 5216.750574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.806177] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5216.810393] Lustre: Skipped 18 previous similar messages [ 5217.678778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.555921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.575318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.412437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.310779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.345385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.317806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.088485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.974893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.710658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.684693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.536108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.523691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.240305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.129621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.934892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.835057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.650797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.647543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.609976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.563632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.422575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.379301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.120838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.044348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.857576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.798141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.730916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.632727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.377540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.169307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.153647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.043061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.831047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.772427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.735961] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5248.739683] Lustre: Skipped 35 previous similar messages [ 5248.752532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5248.762366] Lustre: Skipped 35 previous similar messages [ 5248.767052] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5248.771670] Lustre: Skipped 35 previous similar messages [ 5248.775130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.947801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.968909] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5249.976995] Lustre: Skipped 36 previous similar messages [ 5250.766786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.772532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.672293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.620444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.469261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.273126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.071688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.895167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.714435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.469146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.367558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.072272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.833118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.620296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.329246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.263886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.099533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.824947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.656939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.483673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.256161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.127644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.825788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.730563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.426124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.316572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.101306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.859434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.686292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.638724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.517909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.231324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.978328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.746843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.642941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.368399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.221310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.035200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.866060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.641121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.439637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.197617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.092087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.980108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.778759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.495785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.191460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.899264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.729436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.507288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.341089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.214223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.113993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.009521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.090978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.922350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.690754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.582107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.378690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.299265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.215079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.254309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.113940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.915866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.756853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.629502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.455298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.343361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.160662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.991639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.826771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.661710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.451167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.254892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.994835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.915644] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5312.923981] Lustre: Skipped 76 previous similar messages [ 5312.936493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.945632] Lustre: Skipped 76 previous similar messages [ 5312.952175] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5312.959179] Lustre: Skipped 76 previous similar messages [ 5312.963989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.854686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.794210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.800612] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5314.803177] Lustre: Skipped 77 previous similar messages [ 5315.580988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.589696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.457348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.331442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.197479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.316389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.242046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.191748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.065247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.014690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.981761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.957649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.857602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.832360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.690527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.541810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.298075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.167066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.885946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.808128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.728329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.574956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.237873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.170588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.009380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.055206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.764121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.655197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.503735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.472575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.255814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.120921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.909323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.685904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.461283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.353218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.071879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.836022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.555183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.396735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.239550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.192643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.184971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.194245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.985658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.979599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.796908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.709376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.552434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.593186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.564282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.699816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.701909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.494945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.732827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.765994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.855712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.819968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.783464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.734548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.781699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.793131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.796796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.945324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.109817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.260904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.217505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.159873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.955440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.768474] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5378.775623] LustreError: 172047:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5378.799784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.945064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.154472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.112290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.200841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.031833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.115567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.002078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.946907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.932390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.960321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.989121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.838394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.808178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.887659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.690851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.670250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.613956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.472353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.131905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.844744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.564486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.631791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.432422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.152941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.907334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.728930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.610996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.499429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.374926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.211035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.858173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.764113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.525113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.262568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.022163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.831833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.668333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.486815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.223502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.017566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.724735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.596558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.474408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.266817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.962926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.730254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.451487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.221782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.902867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.710556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.434531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.275634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.129725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.834448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.590255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.569365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.266742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.044479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.889239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.591885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.316563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.078607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.909809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.611541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.306471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.047367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.598398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.323323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.078347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.004965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.898743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.866728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.848830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.653945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.489227] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5441.502237] Lustre: Skipped 146 previous similar messages [ 5441.516721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.538729] Lustre: Skipped 146 previous similar messages [ 5441.580397] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5441.603587] Lustre: Skipped 146 previous similar messages [ 5441.620219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.675971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.806654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.819325] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5443.824772] Lustre: Skipped 146 previous similar messages [ 5444.704944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.698798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.622726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.664546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.745147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.716552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.744574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.719788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.628835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.583923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.631596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.571137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.618863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.675659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.963740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.766242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.632589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.774965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.873117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.729676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.821690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.646888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.448440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.358596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.291053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.029031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.807545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.662576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.500095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.378965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.199027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.049150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.012798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.013659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.928547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.947973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.859272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.897425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.811081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.708545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.650205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.520932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.535251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.546847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.531733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.459748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.215259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.989529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.738443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.415400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.104479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.825923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.773497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.619052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.536729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.348255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.370461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.151341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.083871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.965518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.912667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.016548] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5501.026334] LustreError: 172047:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5501.049552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.905357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.735246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.706981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.590598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.531178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.342351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.133975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.814637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.538674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.353926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.209531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.014184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.879813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.598842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.338359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.183771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.014058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.863222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.665397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.438123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.131952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.007866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.892142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.854788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.657789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.403710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.351347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.148810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.907487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.602362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.389915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.229245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.010764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.795517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.621896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.497983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.307728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.214568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.934265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.758967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.490514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.456982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.298818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.342990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.324958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.293353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.345786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.218132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.201195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.080587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.799405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.591088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.330577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.167772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.984858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.750938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.470887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.171718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.899627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.870353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.637896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.384336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.177109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.075388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.987237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.930673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.671147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.509682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.414526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.214257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.215632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.147787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.018343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.000419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.914499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.968820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.895488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.987531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.814376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.769049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.674392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.724453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.694894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.521507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.538463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.267316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.143299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.976580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.025718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.900710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.897957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.746708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.604111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.368778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.310037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.156247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.006928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.956320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.137431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.122833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.155645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.137898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.089028] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5590.102877] LustreError: 172047:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5590.111088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.930441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.845078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.685420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.651356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.601242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.551912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.446829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.147984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.799137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.473472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.041807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.692539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.413286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.195647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.970544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.701761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.555982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.450915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.112488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.120185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.977627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.004172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.046626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.285429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.303934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.411167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.454589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.195052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.416526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.258249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.265217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.034381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.966649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.968894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.211101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.458600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.500544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.247523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.924478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.607639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.725436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.630964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.470944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.288873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.141370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.954453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.899594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.695186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.547124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.290387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.050867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.766509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.610250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.565649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.373988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.245948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.146953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.893159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.768420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.714204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.628760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.748666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.537084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.482272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.228785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.093526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.702535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.636465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.444116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.367885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.181185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.997710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.775113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.661159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.459441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.452462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.317497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.216314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.061296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.905766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.723412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.523464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.218061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.159102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.132644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.116702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.068613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.003540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.866611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.717536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.544646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.379298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.198069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.223862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.172251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.069163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.842549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.756217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.619674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.524687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.287525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.091630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.722140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.509049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.253737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.002793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.769801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.521881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.355665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.106591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.037538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.755773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.436384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.323971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.229451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.119064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.953207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.961621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.661594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.589186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.304713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.183322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.956624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.775677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.517398] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5697.530816] Lustre: Skipped 291 previous similar messages [ 5697.561829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.574342] Lustre: Skipped 291 previous similar messages [ 5697.579372] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5697.581491] Lustre: Skipped 291 previous similar messages [ 5697.586701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.540341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.357714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.322225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.330630] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5700.333327] Lustre: Skipped 292 previous similar messages [ 5701.308626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.080529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.982684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.957708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.915163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.749689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.508547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.228207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.160982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.927849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.679738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.459342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.237240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.950541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.769975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.868912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.804743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.659828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.682858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.791764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.023141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.953648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.927496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.879654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.949038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.870811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.822685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.474303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.492924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.277555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.075733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.817424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.817329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.784182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.948448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.810252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.879050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.654634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.405358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.034040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.196231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.335045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.229246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.092383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.037175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.228101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.352355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.406310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.584393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.596106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.874375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.944823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.883054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.103410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.278029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.172756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.341858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.265057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.027728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.736234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.751058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.938793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.658627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.622809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.646863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.752712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.994936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.114283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.183326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.328564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.702449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.981722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.133864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.225848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.514887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.329812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.032777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.756358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.641308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.654755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.527351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.818242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.696563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.752687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.956251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.817705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.653264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.743098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.509552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.238954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.056542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.084259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.877181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.881787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.862737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.683163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.822963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.716479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.845745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.508794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.550571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.300808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.335640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.347377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.878344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.927920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.891954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.101446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.345750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.626331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.789779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.683889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.721612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.786281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.886768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.799581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.859617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.863595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.570389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.143673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.903935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.881843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.653280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.545445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.375336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.182831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.987939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.738856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.718365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.571595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.669078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.614761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.650920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.503498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.538545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.945466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.983464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.821341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.159154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.345319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.394902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.501872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.585756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.837567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.710773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.741647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.635853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.681706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.539819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.605386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.514764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.532504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.560233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.670464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.613068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.666876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.833439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.636761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.774256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.699818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.702712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.721513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.813316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.604302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.675211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.652569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.678328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.540869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.393689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.174701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.840559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.637207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.433904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.237639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.257190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.207668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.404567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.130274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.010322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.165374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.187534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.096764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.899130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.009625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.866508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.777528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.596476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.498956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.320166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.213455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.119795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.196650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.152873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.261911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.281473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.395959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.418878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.568355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.731868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.846502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.879086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.914177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.970467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.916565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.701942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.772383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.686686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.747553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.619324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.548537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.604924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.633554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.677566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.522532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.449025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.455666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.502239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.473832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.388057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.313824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.422115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.358737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.396255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.335210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.393259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.180710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.088231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.117931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.995879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.063511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.853068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.715274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.522280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.518337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.405151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.617907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.372710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.412961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.180784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.258465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.214585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.331774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.329516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.444666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.266950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.076674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.929935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.924260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.878321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.774495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.776430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.765339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.660077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.561506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.441237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.363613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.360791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.477919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.420261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.575397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.638994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.575386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.567989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.644443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.599777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.508628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.616901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.590698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.407076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.253435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.156084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.176642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.181651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.231229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.015361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.909924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.713911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.731866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.558100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.550796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.344778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.299949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.071279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.080655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.847721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.667665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.403159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.206447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.109855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.879265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.791641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.767638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.451764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.473140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.356866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.331873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.337028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.321239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.461424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.369830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.145375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.000633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.040603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.874906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.994210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.898174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.968823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.895850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.090754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.230942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.292320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.151241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.286148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.179595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.124081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.264713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.406469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.446612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.592499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.446168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.450623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.521617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.440032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.385851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.518930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.739700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.733936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.705441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.785854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.776029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.731410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.509944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.539314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.387430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.336320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.409400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.292297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.626446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.629742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.445977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.382747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.285692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.173964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.105883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.936555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.857729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.831877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.869314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.942415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.788579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.857808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.871831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.865069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.015919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.117812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.367320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.498502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.530039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.630277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.569354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.661375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.612815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.654824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.765599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.569965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.605348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.742278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.701535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.385396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.516399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.460866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.476720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.395194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.355299] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6063.369673] LustreError: 172047:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6063.375685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.326666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.545996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.701483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.649985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.727304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.634181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.638667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.559553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.542882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.570546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.647182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.660976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.634360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.637084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.605269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.629649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.628241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.884147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.721347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.892976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.867301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.889231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.800064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.832378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.833713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.732392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.604707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.608319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.708919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.668212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.649132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.037114] LustreError: 173847:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6407.060824] LustreError: 172047:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6799.843416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6799.847046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.851551] Lustre: Skipped 408 previous similar messages [ 6799.854749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.870542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.870617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.214328] Lustre: server umount lustre-OST0000 complete [ 6805.989342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6805.994168] Lustre: Skipped 2 previous similar messages [ 6808.363451] Lustre: server umount lustre-MDT0000 complete [ 6811.108240] LustreError: 172028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.133424] LustreError: 172028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6813.033698] LustreError: 172007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658782 with bad export cookie 17598596609392178534 [ 6813.037848] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6813.043899] LustreError: 172007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6813.536829] Lustre: server umount lustre-MDT0001 complete [ 6826.365577] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 6829.830342] Key type lgssc unregistered [ 6830.129912] LNet: 227219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6830.137359] LNetError: 227219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6830.156282] LNet: Removed LNI 192.168.201.157@tcp [ 6831.045163] Key type .llcrypt unregistered [ 6831.046924] Key type ._llcrypt unregistered [ 6853.501675] Key type ._llcrypt registered [ 6853.506544] Key type .llcrypt registered [ 6853.586945] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 6854.377840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6854.479887] alg: No test for adler32 (adler32-zlib) [ 6855.517211] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 6855.762836] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 6857.407245] Key type lgssc registered [ 6858.512550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6864.990628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6865.015757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.366458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6870.667747] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6876.651595] LustreError: 228629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.669263] LustreError: 228629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6878.231113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.564125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6883.132992] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6892.176208] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6898.624821] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6905.174682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6905.609351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.566929] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6911.826595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6920.150256] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6928.761574] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6930.558559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.569257] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6930.573905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6930.578869] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6930.579800] LustreError: 228649:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6932.431220] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 6933.473362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.481937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.485699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6937.563400] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 6937.566899] Lustre: Skipped 1 previous similar message [ 6938.533899] Lustre: server umount lustre-OST0000 complete [ 6950.671164] LustreError: 228649:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6973.435970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.776243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6980.287740] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6987.050752] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6989.592022] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6996.665258] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88cb4c70d800.ost_server_uuid 50 [ 6998.171270] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88cb4c70d800.ost_server_uuid in FULL state after 0 sec [ 6998.581885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6998.601452] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6998.604897] LustreError: 230319:0:(lod_qos.c:1827:lod_ost_alloc_specific()) can't lstripe objid [0x2000013a1:0x3:0x0]: have 0 want 1 [ 6998.605892] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7000.764365] Lustre: DEBUG MARKER: conf-sanity test_101b: @@@@@@ FAIL: Can't create file [ 7008.716731] Lustre: DEBUG MARKER: SKIP: conf-sanity test_102 skipping excluded test 102 [ 7010.593798] Lustre: DEBUG MARKER: == conf-sanity test 103: rename filesystem name ========== 21:42:58 (1788658978) [ 7041.701365] LustreError: 228649:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006ca76fa4 x1875544715177216/t0(0) o7->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 264/240 e 0 to 0 dl 0 ref 2 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7042.528180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7042.531837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.543144] Lustre: Skipped 1 previous similar message [ 7042.563102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.579908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.027665] Lustre: server umount lustre-MDT0000 complete [ 7049.709773] LustreError: 228629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.742687] LustreError: 228629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7051.511721] LustreError: 233187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659020 with bad export cookie 6933245453585415417 [ 7051.522220] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7051.528841] LustreError: 233187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7051.939526] Lustre: server umount lustre-MDT0001 complete [ 7065.239425] Lustre: server umount lustre-OST0000 complete [ 7092.354518] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7103.031157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.532812] LustreError: 235299:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.591075] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.519852] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7109.093182] LustreError: 235300:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7113.190681] LustreError: 235299:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.233888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.791333] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7122.904448] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7126.745586] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7134.979313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.633093] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7142.831617] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7147.896539] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49728) [ 7151.013917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.185588] Lustre: mylustre-OST0001: new disk, initializing [ 7151.193052] Lustre: srv-mylustre-OST0001: No data found on store. Initialize space. [ 7151.198306] Lustre: 237609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7157.849639] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7158.355708] Lustre: ctl-mylustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7158.366431] Lustre: cli-mylustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7158.492940] Lustre: mylustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7166.402624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7171.958554] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7184.354052] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.361304] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7184.362038] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.362043] Lustre: Skipped 3 previous similar messages [ 7184.366560] Lustre: Skipped 3 previous similar messages [ 7187.707432] Lustre: server umount mylustre-MDT0000 complete [ 7189.476614] LustreError: 235295:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.507807] LustreError: 235295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7191.192529] LustreError: 235276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659160 with bad export cookie 6933245453585418217 [ 7191.195893] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7191.212845] LustreError: 235276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7191.517279] Lustre: server umount mylustre-MDT0001 complete [ 7205.633264] Lustre: server umount mylustre-OST0000 complete [ 7219.946433] Lustre: server umount mylustre-OST0001 complete [ 7240.905421] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7252.509337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.995729] LustreError: 240748:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.099647] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.105550] Lustre: Skipped 1 previous similar message [ 7258.806839] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7267.791992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.968252] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7276.186915] Lustre: 241890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7282.584948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7289.787261] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7293.443755] LustreError: 242245:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.481272] LustreError: 242245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7299.148823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.557251] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7300.557793] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49760) [ 7305.749690] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7313.242230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7317.675857] Lustre: 243762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7326.184464] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.200391] Lustre: Skipped 1 previous similar message [ 7326.297567] Lustre: server umount tfs-MDT0000 complete [ 7329.777829] LustreError: 240723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659298 with bad export cookie 6933245453585420149 [ 7329.785444] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7329.788988] LustreError: 240723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7330.189865] Lustre: server umount tfs-MDT0001 complete [ 7344.295595] Lustre: server umount tfs-OST0000 complete [ 7347.423101] Lustre: 227940:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788659300/real 1788659300] req@ffff9424cb7c8000 x1875544715305088/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788659316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7347.465997] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.476487] Lustre: Skipped 3 previous similar messages [ 7348.917354] Lustre: server umount tfs-OST0001 complete [ 7373.267638] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7385.165299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.722143] LustreError: 246010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.740063] LustreError: 246010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7385.851290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.858092] Lustre: Skipped 3 previous similar messages [ 7391.512924] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7401.767620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.505173] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7410.451567] Lustre: 247152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7417.355457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7425.845525] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7430.085587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49792) [ 7434.454054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.878679] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 7441.432472] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7451.172470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7455.242762] Lustre: 249026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7463.167383] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 21:50:30 (1788659430) [ 7484.385849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7484.397852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7484.408191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7484.410818] Lustre: Skipped 3 previous similar messages [ 7487.671188] Lustre: server umount lustre-MDT0000 complete [ 7491.162683] LustreError: 247154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659460 with bad export cookie 6933245453585421815 [ 7491.167366] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7491.170510] LustreError: 247154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7497.805702] Lustre: server umount lustre-MDT0001 complete [ 7508.452447] Lustre: server umount lustre-OST0000 complete [ 7518.833409] Lustre: server umount lustre-OST0001 complete [ 7525.803538] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 7536.112570] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7575.383463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7575.605633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7575.625588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7575.694689] Lustre: lustre-MDT0000: new disk, initializing [ 7575.778261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7575.784639] Lustre: Skipped 3 previous similar messages [ 7575.797936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7581.007830] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7592.263572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7592.428807] Lustre: 252211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7592.486533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7592.490548] Lustre: Skipped 1 previous similar message [ 7592.573177] Lustre: lustre-MDT0001: new disk, initializing [ 7592.736465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7592.755204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7597.183844] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7607.660820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.000569] Lustre: lustre-OST0000: new disk, initializing [ 7608.007750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7608.017426] Lustre: 253788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7610.011504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7610.017585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7610.186504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7614.827104] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7625.207367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7625.438580] Lustre: lustre-OST0001: new disk, initializing [ 7625.444491] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7625.456049] Lustre: 254661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7627.419247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7627.451265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7627.575212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7632.624412] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7645.281511] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 21:53:32 (1788659612) [ 7660.082766] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 21:53:47 (1788659627) [ 7676.900862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7676.904977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.909699] LustreError: Skipped 1 previous similar message [ 7676.926881] Lustre: Skipped 5 previous similar messages [ 7676.933842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7676.936531] Lustre: Skipped 6 previous similar messages [ 7681.344409] Lustre: server umount lustre-OST0000 complete [ 7684.069824] LustreError: 253773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.095572] LustreError: 253773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7685.088974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7691.358048] Lustre: server umount lustre-MDT0000 complete [ 7695.298297] LustreError: 252201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659664 with bad export cookie 6933245453585423362 [ 7695.304504] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.307263] LustreError: 252201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7701.766667] Lustre: server umount lustre-MDT0001 complete [ 7714.703193] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 7787.629608] Lustre: server umount lustre-OST0001 complete [ 7793.818806] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 7810.759479] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7850.805081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7851.098874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7851.160985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7851.340772] Lustre: lustre-MDT0000: new disk, initializing [ 7851.496250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.513435] Lustre: Skipped 3 previous similar messages [ 7851.551424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7856.189655] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7866.499226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.587728] Lustre: 259520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7866.785819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7871.442573] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7882.815463] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7888.774230] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7896.213900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7896.607463] Lustre: lustre-OST0000: new disk, initializing [ 7896.612732] Lustre: Skipped 1 previous similar message [ 7896.618707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7896.621697] Lustre: Skipped 2 previous similar messages [ 7896.627473] Lustre: 261487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7897.731741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7897.737913] Lustre: Skipped 1 previous similar message [ 7897.742236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7897.895727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7903.545133] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7915.646786] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7923.687782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7923.705424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.738307] Lustre: Skipped 5 previous similar messages [ 7923.758886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7923.767622] Lustre: Skipped 8 previous similar messages [ 7929.609749] Lustre: server umount lustre-OST0000 complete [ 7942.772756] Lustre: server umount lustre-MDT0000 complete [ 7944.161970] LustreError: 260390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7944.197312] LustreError: 260390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7947.327120] LustreError: 259510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659916 with bad export cookie 6933245453585424839 [ 7947.328533] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.333948] LustreError: 259510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7947.941590] Lustre: server umount lustre-MDT0001 complete [ 7961.569883] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 7965.236409] Key type lgssc unregistered [ 7965.736062] LNet: 263247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7965.747202] LNetError: 263247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7965.793935] LNet: Removed LNI 192.168.201.157@tcp [ 7967.021187] Key type .llcrypt unregistered [ 7967.029493] Key type ._llcrypt unregistered [ 7998.777202] Key type ._llcrypt registered [ 7998.780766] Key type .llcrypt registered [ 7998.887981] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8000.267516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8000.292158] alg: No test for adler32 (adler32-zlib) [ 8001.371749] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 8001.618518] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8003.279223] Key type lgssc registered [ 8004.895211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8013.731932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8013.750829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.390441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.187603] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8025.577869] LustreError: 264709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8025.619741] LustreError: 264709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8030.443149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.699968] LustreError: 264708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.106874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8036.896126] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8045.753144] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8052.357669] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8060.181736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8060.811582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.571469] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8077.296656] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8078.525986] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8081.384201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.395336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.409192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8083.333350] Lustre: server umount lustre-OST0000 complete [ 8090.594272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8090.617934] Lustre: Skipped 1 previous similar message [ 8090.618187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8090.631733] Lustre: Skipped 1 previous similar message [ 8095.719226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8095.730624] Lustre: Skipped 1 previous similar message [ 8099.811199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8099.826624] Lustre: Skipped 1 previous similar message [ 8102.368346] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8102.636933] Lustre: server umount lustre-MDT0000 complete [ 8104.949099] LustreError: 264708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.975471] LustreError: 264708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8106.505035] LustreError: 264690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660075 with bad export cookie 15978039930992533230 [ 8106.533577] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8106.912039] Lustre: server umount lustre-MDT0001 complete [ 8120.417252] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8124.155768] Key type lgssc unregistered [ 8124.534508] LNet: 268305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8124.544515] LNetError: 268305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8125.606600] LNet: Removed LNI 192.168.201.157@tcp [ 8126.910975] Key type .llcrypt unregistered [ 8126.913163] Key type ._llcrypt unregistered [ 8149.914639] Key type ._llcrypt registered [ 8149.916750] Key type .llcrypt registered [ 8150.036976] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8151.904435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.985123] alg: No test for adler32 (adler32-zlib) [ 8153.232958] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 8153.643738] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8155.295216] Key type lgssc registered [ 8156.368956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8166.686578] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8166.719066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.303528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8187.423878] LustreError: 269498:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.157@tcp: failed processing log, type 4: rc = -110 [ 8214.116305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8221.286820] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8229.521881] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8231.656780] Lustre: server umount lustre-OST0000 complete [ 8249.818913] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8253.998533] Key type lgssc unregistered [ 8254.330721] LNet: 271129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8254.340480] LNetError: 271129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8254.354982] LNet: Removed LNI 192.168.201.157@tcp [ 8255.421245] Key type .llcrypt unregistered [ 8255.425369] Key type ._llcrypt unregistered [ 8302.518405] Key type ._llcrypt registered [ 8302.520681] Key type .llcrypt registered [ 8302.664388] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8303.872692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8303.950064] alg: No test for adler32 (adler32-zlib) [ 8305.157608] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 8305.380632] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8307.063185] Key type lgssc registered [ 8308.574900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8323.003881] loop0: detected capacity change from 0 to 204800000 [ 8323.239100] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8323.252106] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8323.591151] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8324.732798] loop0: detected capacity change from 0 to 204800000 [ 8324.929069] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8324.950037] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8325.417456] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8326.520964] loop0: detected capacity change from 0 to 204800000 [ 8326.686069] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8326.695391] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8326.790211] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8327.836315] loop0: detected capacity change from 0 to 204800000 [ 8328.029805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8329.114249] loop0: detected capacity change from 0 to 204800000 [ 8329.152248] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8333.333249] loop0: detected capacity change from 0 to 204800000 [ 8333.388977] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8337.604396] loop0: detected capacity change from 0 to 204800000 [ 8337.621285] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8342.045295] loop0: detected capacity change from 0 to 204800000 [ 8342.083085] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8346.567194] loop0: detected capacity change from 0 to 204800000 [ 8346.715510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8346.797415] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.884776] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8346.893965] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8346.919897] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8346.938143] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8346.959225] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8346.973944] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8346.987077] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8347.012135] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8347.027507] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8347.057095] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8347.072992] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8347.092637] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8347.102558] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8347.121638] Lustre: 274561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8354.189741] Lustre: server umount MGS complete [ 8355.293984] loop0: detected capacity change from 0 to 204800000 [ 8355.366151] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.157811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.215064] Lustre: lustre-MDT0000: Aborting client recovery [ 8357.380935] loop1: detected capacity change from 0 to 204800000 [ 8357.663782] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8357.815427] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8357.843535] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8357.860792] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8357.892275] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8357.900704] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8357.911577] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8357.930585] Lustre: 274994:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8357.995264] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8358.314400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8358.413599] Lustre: lustre-MDT0001: Aborting client recovery [ 8359.703716] loop2: detected capacity change from 0 to 204800000 [ 8359.984709] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8360.101340] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8360.123521] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8360.145615] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8360.177461] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8360.206838] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8360.231057] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8360.246228] Lustre: 275073:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8360.393680] Lustre: Found index 0 for lustre-OST0000, updating log [ 8360.447476] Lustre: 275091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8360.513058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.544396] Lustre: lustre-OST0000: Aborting recovery [ 8361.630256] loop3: detected capacity change from 0 to 204800000 [ 8361.891062] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8361.978053] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8361.993267] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8362.003854] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8362.030702] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8362.051839] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8362.078421] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8362.088637] Lustre: 275156:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8362.118781] Lustre: Found index 1 for lustre-OST0001, updating log [ 8362.226203] Lustre: 275162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8362.338155] Lustre: lustre-OST0001: Aborting recovery [ 8366.384570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8366.404136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8366.431395] LustreError: 274937:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8366.460403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8366.509056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8368.432570] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8368.432987] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8372.385137] LustreError: 274904:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 962us: rc = -2 [ 8372.895237] LustreError: 275395:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 4111us: rc = -2 [ 8372.917381] LustreError: 275395:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 24 previous similar messages [ 8389.088186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.089840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8389.090217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.096768] Lustre: Skipped 2 previous similar messages [ 8394.208457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8394.214317] Lustre: Skipped 6 previous similar messages [ 8399.329596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8399.335405] Lustre: Skipped 3 previous similar messages [ 8400.351189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8400.565533] Lustre: server umount lustre-MDT0000 complete [ 8401.445059] LustreError: 274888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660370 with bad export cookie 2712910656835973361 [ 8401.466045] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8404.449245] LustreError: 274902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.450324] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.460132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8404.461122] LustreError: 274902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8404.514158] Lustre: Skipped 2 previous similar messages [ 8406.497738] LustreError: 274902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.517881] LustreError: 274902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8407.850165] Lustre: server umount lustre-MDT0001 complete [ 8423.392029] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.540837] Lustre: server umount lustre-OST0000 complete [ 8430.898513] Lustre: server umount lustre-OST0001 complete [ 8439.773607] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 22:06:47 (1788660407) [ 8470.804385] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 8482.556404] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8516.840090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.123775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8517.158452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8517.250167] Lustre: lustre-MDT0000: new disk, initializing [ 8517.371966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.376755] Lustre: Skipped 1 previous similar message [ 8517.392581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8521.826121] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8533.626776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8533.874458] Lustre: 279231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8533.939590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8533.942672] Lustre: Skipped 1 previous similar message [ 8534.035588] Lustre: lustre-MDT0001: new disk, initializing [ 8534.141194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8534.196351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8534.206263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8539.689591] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8548.818442] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8554.869066] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8562.080255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8562.278349] Lustre: lustre-OST0000: new disk, initializing [ 8562.284576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8562.292665] Lustre: 281201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8562.430915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.618182] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8569.469915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8569.486449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8569.578313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8577.842736] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8583.372099] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8597.686951] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8601.571420] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8609.474935] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8651.746622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8651.749496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.759117] LustreError: Skipped 1 previous similar message [ 8651.768237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8651.779372] Lustre: Skipped 1 previous similar message [ 8651.806422] Lustre: Skipped 4 previous similar messages [ 8654.818338] Lustre: server umount lustre-OST0000 complete [ 8661.984620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.991224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8662.005273] Lustre: Skipped 1 previous similar message [ 8662.016651] Lustre: Skipped 1 previous similar message [ 8665.041886] Lustre: server umount lustre-MDT0000 complete [ 8667.104916] LustreError: 282231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8667.122906] LustreError: 282231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8668.761467] LustreError: 279224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660637 with bad export cookie 2712910656835976077 [ 8668.768798] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8668.773362] LustreError: 279224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8669.149931] Lustre: server umount lustre-MDT0001 complete [ 8676.317366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.137443] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8685.445791] Lustre: server umount MGS complete [ 8697.934486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.212704] LustreError: 285071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.227789] LustreError: 285084:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8698.263453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8702.780157] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8703.467568] LustreError: 285072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8711.184290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.400291] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8722.207334] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8727.091578] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8733.441974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8733.759312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.775137] Lustre: Skipped 1 previous similar message [ 8739.619760] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8750.206692] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8783.330653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8783.345361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.360555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.299210] Lustre: server umount lustre-OST0000 complete [ 8793.575975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8793.582600] LustreError: Skipped 1 previous similar message [ 8793.588607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.616146] Lustre: Skipped 1 previous similar message [ 8806.368116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8806.574895] Lustre: server umount lustre-MDT0000 complete [ 8809.443812] LustreError: 285068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.461779] LustreError: 285068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8810.809452] LustreError: 285052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660780 with bad export cookie 2712910656835978016 [ 8810.823703] LustreError: 285052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8810.824161] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8811.266718] Lustre: server umount lustre-MDT0001 complete [ 8824.506637] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8827.148219] Key type lgssc unregistered [ 8827.590202] LNet: 289019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8827.612752] LNetError: 289019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8827.637200] LNet: Removed LNI 192.168.201.157@tcp [ 8828.523642] Key type .llcrypt unregistered [ 8828.526682] Key type ._llcrypt unregistered [ 8868.618889] Key type ._llcrypt registered [ 8868.620895] Key type .llcrypt registered [ 8868.722737] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 8883.529464] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8885.032901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8885.048349] alg: No test for adler32 (adler32-zlib) [ 8886.275541] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 8886.595659] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8888.344434] Key type lgssc registered [ 8889.782354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8924.724542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8924.760476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.053188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8926.111948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8926.188151] Lustre: lustre-MDT0000: new disk, initializing [ 8926.288446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.313533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8930.943704] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8941.389208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.490692] Lustre: 292975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8941.529280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8941.532722] Lustre: Skipped 1 previous similar message [ 8941.636350] Lustre: lustre-MDT0001: new disk, initializing [ 8941.739100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8941.789286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8941.809104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8947.032752] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8955.244572] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8960.886603] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8968.107741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.414365] Lustre: lustre-OST0000: new disk, initializing [ 8968.419692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8968.428940] Lustre: 294942:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8968.516743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.597729] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8978.516902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8978.528513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8978.654630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8983.256537] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8990.229944] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9000.050454] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9010.083266] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9018.711622] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9060.833154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9060.835172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.837547] LustreError: Skipped 1 previous similar message [ 9060.846314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9060.847686] Lustre: Skipped 1 previous similar message [ 9060.859108] Lustre: Skipped 1 previous similar message [ 9065.958914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9065.966878] Lustre: Skipped 1 previous similar message [ 9066.807233] Lustre: server umount lustre-OST0000 complete [ 9071.077497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9071.092620] Lustre: Skipped 1 previous similar message [ 9071.098612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9076.199802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9076.204828] Lustre: Skipped 2 previous similar messages [ 9076.607697] Lustre: server umount lustre-MDT0000 complete [ 9080.628951] LustreError: 294941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661049 with bad export cookie 15180090109811920187 [ 9080.636432] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9080.641204] LustreError: 294941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9081.172170] Lustre: server umount lustre-MDT0001 complete [ 9090.256303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.883944] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9099.842945] Lustre: server umount MGS complete [ 9111.862524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.498479] LustreError: 298875:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9112.574215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.833945] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9117.669151] LustreError: 298863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.683814] LustreError: 298863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9122.794679] LustreError: 298862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.718792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9125.035735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9129.516182] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9136.350841] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9140.717647] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9145.390927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9145.621655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9151.129813] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9158.486073] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9186.788140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9186.792905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9186.802221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.836077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.836496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.075986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.084290] Lustre: Skipped 4 previous similar messages [ 9200.095192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9200.361591] Lustre: server umount lustre-OST0000 complete [ 9204.192663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.409739] Lustre: server umount lustre-MDT0000 complete [ 9214.131310] LustreError: 300708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661183 with bad export cookie 15180090109811922140 [ 9214.140680] LustreError: 300708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9214.141388] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.479949] Lustre: server umount lustre-MDT0001 complete [ 9225.550570] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 9228.082992] Key type lgssc unregistered [ 9228.395584] LNet: 302806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9228.404793] LNetError: 302806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9228.422639] LNet: Removed LNI 192.168.201.157@tcp [ 9229.233098] Key type .llcrypt unregistered [ 9229.234551] Key type ._llcrypt unregistered [ 9246.914669] Key type ._llcrypt registered [ 9246.917215] Key type .llcrypt registered [ 9246.996525] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 9260.352188] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9261.377852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9261.475609] alg: No test for adler32 (adler32-zlib) [ 9262.458608] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 9262.617145] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 9264.239274] Key type lgssc registered [ 9265.185171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9286.230958] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9299.114348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9299.137872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.385826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9300.405415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9300.475959] Lustre: lustre-MDT0000: new disk, initializing [ 9300.544275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9300.560144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9304.622405] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9313.946376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.167704] Lustre: lustre-OST0000: new disk, initializing [ 9314.171358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9314.174892] Lustre: Skipped 1 previous similar message [ 9314.184438] Lustre: 306846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9314.261461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.747800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9317.767782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9317.806400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9320.321677] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10090.981419] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10090.992081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10093.146587] Lustre: server umount lustre-MDT0000 complete [10096.600266] LustreError: 306031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662065 with bad export cookie 11389174410525538666 [10096.603152] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10096.609185] LustreError: 306031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10096.944896] Lustre: server umount lustre-OST0000 complete [10119.818664] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [10129.995301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10159.627367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.809456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10159.835807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10159.906324] Lustre: lustre-MDT0000: new disk, initializing [10159.975782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10159.999734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10163.949421] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10173.382891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.469785] Lustre: 311170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10173.511774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10173.516995] Lustre: Skipped 1 previous similar message [10173.580540] Lustre: lustre-MDT0001: new disk, initializing [10173.703588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10173.746613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10173.757939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10178.777982] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10187.391757] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10192.132578] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10198.609668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.807721] Lustre: lustre-OST0000: new disk, initializing [10198.814158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10198.817839] Lustre: 313139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10198.885964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10204.865381] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10205.231835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10205.246053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10205.434896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10214.300588] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10219.142951] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10219.356511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10223.796446] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10223.979362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10226.152195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10226.154788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10226.167971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10226.177235] Lustre: Skipped 1 previous similar message [10231.284851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10231.291862] Lustre: Skipped 1 previous similar message [10231.851257] Lustre: server umount lustre-OST0000 complete [10236.390709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.408781] Lustre: Skipped 1 previous similar message [10236.415277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10241.507620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10241.521991] Lustre: Skipped 2 previous similar messages [10241.904852] Lustre: server umount lustre-MDT0000 complete [10245.321590] LustreError: 311163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662214 with bad export cookie 11389174410528479485 [10245.322135] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10245.330417] LustreError: 311163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10245.656546] Lustre: server umount lustre-MDT0001 complete [10248.392939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10249.690672] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 22:36:57 (1788662217) [10255.362663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10255.824735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10260.223518] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10260.967029] LustreError: 315450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.979180] LustreError: 315450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10265.059179] LustreError: 315451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10268.700840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.030788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10272.820571] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10279.033049] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10283.652693] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10290.114546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10290.392391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10296.674660] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10303.723891] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10310.376332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10310.505896] Lustre: lustre-OST0001: new disk, initializing [10310.508929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10310.513671] Lustre: 318285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10310.588917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10315.921539] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10316.332365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10316.337784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10316.374236] LustreError: 318663:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10317.423937] LustreError: 318663:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10317.446083] LustreError: 318663:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10318.507169] LustreError: 318663:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10318.527643] LustreError: 318663:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10320.548308] LustreError: 318670:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10320.557025] LustreError: 318670:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10324.657312] LustreError: 318670:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10324.674748] LustreError: 318670:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10324.690756] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10324.897522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10329.751429] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88cb43786000.ost_server_uuid 50 [10331.240111] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88cb43786000.ost_server_uuid in FULL state after 0 sec [10332.969105] LustreError: 318670:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10332.984953] LustreError: 318670:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10337.498835] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88cb43786000.ost_server_uuid 50 [10339.523551] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88cb43786000.ost_server_uuid in FULL state after 0 sec [10341.331372] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10356.679577] Lustre: lustre-OST0001: Not available for connect from 192.168.201.57@tcp (stopping) [10356.707511] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10356.711576] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10356.721067] LustreError: Skipped 2 previous similar messages [10356.737512] Lustre: Skipped 2 previous similar messages [10362.113833] Lustre: server umount lustre-OST0001 complete [10362.344795] LustreError: 317516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.374346] LustreError: 317516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10367.467926] LustreError: 317272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.492438] LustreError: 317272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10371.554964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10371.562409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.578760] LustreError: 317516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.588298] LustreError: 317516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10375.057623] Lustre: server umount lustre-OST0000 complete [10381.805638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10381.817725] LustreError: Skipped 1 previous similar message [10381.823327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.850217] Lustre: Skipped 1 previous similar message [10381.868966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10381.871649] Lustre: Skipped 8 previous similar messages [10385.869322] Lustre: server umount lustre-MDT0000 complete [10387.937674] LustreError: 315450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10387.962431] LustreError: 315450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10389.811961] LustreError: 315430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662359 with bad export cookie 11389174410528480367 [10389.814809] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10389.821342] LustreError: 315430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10390.218987] Lustre: server umount lustre-MDT0001 complete [10401.277493] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10403.924884] Key type lgssc unregistered [10404.187940] LNet: 320972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10404.194874] LNetError: 320972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10404.205744] LNet: Removed LNI 192.168.201.157@tcp [10405.145593] Key type .llcrypt unregistered [10405.147643] Key type ._llcrypt unregistered [10433.551578] Key type ._llcrypt registered [10433.553369] Key type .llcrypt registered [10433.781350] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10435.146775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10435.188657] alg: No test for adler32 (adler32-zlib) [10436.708379] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [10436.958074] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10438.696487] Key type lgssc registered [10440.178821] Lustre: Echo OBD driver; http://www.lustre.org/ [10449.556581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10449.582483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.121760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10456.865352] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10461.668770] LustreError: 322393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.693239] LustreError: 322393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10464.694854] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10466.787228] LustreError: 322394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.618851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.988150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10478.036767] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10489.012607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10489.529963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.402864] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10496.675865] LustreError: 324073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.708286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10503.155785] LustreError: 324075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10503.171456] LustreError: 324075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10505.386908] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10508.269659] LustreError: 324076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.298563] LustreError: 324076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10513.322406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.690068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10519.062974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10521.116562] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10529.188668] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10536.665241] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10536.869454] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10641.892199] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.895763] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10641.928208] LustreError: Skipped 1 previous similar message [10641.931874] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10641.942093] Lustre: Skipped 1 previous similar message [10647.012901] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10647.028500] Lustre: Skipped 1 previous similar message [10652.145139] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10654.175147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10654.285977] Lustre: server umount lustre-OST0001 complete [10657.255391] LustreError: 324075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.278325] LustreError: 324075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10665.952643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10665.961827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10665.971632] Lustre: Skipped 1 previous similar message [10665.975571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10665.978440] Lustre: Skipped 1 previous similar message [10666.981642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10666.990064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.851652] Lustre: server umount lustre-OST0000 complete [10676.705829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10676.712849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.730336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.732877] Lustre: Skipped 3 previous similar messages [10679.639828] Lustre: server umount lustre-MDT0000 complete [10682.866182] LustreError: 322394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.895868] LustreError: 322394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10683.975498] LustreError: 322375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662653 with bad export cookie 6487053974789836277 [10683.980883] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10683.988910] LustreError: 322375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10684.631879] Lustre: server umount lustre-MDT0001 complete [10697.225938] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10699.746270] Key type lgssc unregistered [10700.043734] LNet: 327336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10700.052232] LNetError: 327336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10700.076139] LNet: Removed LNI 192.168.201.157@tcp [10701.209642] Key type .llcrypt unregistered [10701.211981] Key type ._llcrypt unregistered [10753.279800] Key type ._llcrypt registered [10753.281151] Key type .llcrypt registered [10753.498699] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10754.871540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10754.888765] alg: No test for adler32 (adler32-zlib) [10756.298094] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [10756.574136] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10758.391222] Key type lgssc registered [10759.627598] Lustre: Echo OBD driver; http://www.lustre.org/ [10769.831737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10769.842365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.368097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10775.998667] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10781.677808] LustreError: 329626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.708513] LustreError: 329626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10786.512298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10786.810604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10793.008344] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10796.711389] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10803.950516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10804.497777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10810.821786] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10811.766731] LustreError: 331114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10815.863791] LustreError: 331518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10815.889829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10815.893335] LustreError: 331518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10819.845766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10820.145836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10825.230033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10827.149161] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10837.137460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10845.945290] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10861.051810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10861.058067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10861.080915] Lustre: Skipped 3 previous similar messages [10861.095644] Lustre: Skipped 3 previous similar messages [10866.147146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10866.149554] Lustre: Skipped 3 previous similar messages [10866.573953] Lustre: server umount lustre-MDT0000 complete [10870.429468] LustreError: 329605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662839 with bad export cookie 13011397922974039254 [10870.434991] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10870.439860] LustreError: 329605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10870.957896] Lustre: server umount lustre-MDT0001 complete [10885.657518] Lustre: server umount lustre-OST0000 complete [10887.647302] Lustre: 328887:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788662840/real 1788662840] req@ffff9424d6576300 x1875548805596160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788662856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10887.668654] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.470811] Lustre: server umount lustre-OST0001 complete [10907.087378] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10918.512714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.845419] LustreError: 334888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.940323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10923.186609] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10924.012885] LustreError: 334889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.404638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.113131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10938.931393] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10942.082655] Lustre: 336031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10949.765239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10950.231109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10956.193623] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10958.442738] LustreError: 336384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.466435] LustreError: 336384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10964.256103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10965.747736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10965.748575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10971.038975] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10978.863464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10983.007696] Lustre: 337901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10989.567073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10989.585409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.619162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10991.588779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10991.606270] Lustre: Skipped 2 previous similar messages [10994.869803] Lustre: server umount lustre-MDT0000 complete [10996.716372] LustreError: 334889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.742455] LustreError: 334889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10999.994316] LustreError: 334870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662969 with bad export cookie 13011397922974040955 [10999.998202] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11000.017906] LustreError: 334870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11000.566188] Lustre: server umount lustre-MDT0001 complete [11015.381674] Lustre: server umount lustre-OST0000 complete [11018.209616] Lustre: 328886:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788662971/real 1788662971] req@ffff9424cbb63800 x1875548805647488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788662987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11018.247316] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.582237] Lustre: server umount lustre-OST0001 complete [11023.619459] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11025.814032] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11027.464587] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 22:49:55 (1788662995) [11050.039910] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11062.577347] loop0: detected capacity change from 0 to 1099511627776 [11063.314525] SGI XFS with ACLs, security attributes, quota, no debug enabled [11063.332070] *** VALIDATE xfs *** [11063.356389] XFS (loop0): Mounting V5 Filesystem [11063.409333] XFS (loop0): Ending clean mount [11064.743534] loop1: detected capacity change from 0 to 18691697672192 [11171.877424] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [11179.977511] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11215.079181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.317059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11215.335540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11215.396381] Lustre: lustre-MDT0000: new disk, initializing [11215.441729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.445649] Lustre: Skipped 1 previous similar message [11215.467899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11221.215346] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11231.669175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.811967] Lustre: 343038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11231.880297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11231.887282] Lustre: Skipped 1 previous similar message [11231.995896] Lustre: lustre-MDT0001: new disk, initializing [11232.124120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11232.133473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11236.314915] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11245.668285] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11252.858188] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11261.379051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.750913] Lustre: lustre-OST0000: new disk, initializing [11261.757779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11261.764977] Lustre: 345006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11263.816240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11263.826619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11263.998725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11269.638822] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11280.489323] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11286.167370] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11286.449992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11291.583231] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11291.884370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11294.689137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11294.700404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.703839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11294.715930] Lustre: Skipped 1 previous similar message [11294.721432] Lustre: Skipped 4 previous similar messages [11299.819631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11299.827622] Lustre: Skipped 2 previous similar messages [11300.138458] Lustre: server umount lustre-OST0000 complete [11303.909489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11303.918712] LustreError: Skipped 1 previous similar message [11309.904628] Lustre: server umount lustre-MDT0000 complete [11310.048159] LustreError: 343839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.064323] LustreError: 343839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11313.689340] LustreError: 343030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788663282 with bad export cookie 13011397922974042509 [11313.697859] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11313.701538] LustreError: 343030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11314.076748] Lustre: server umount lustre-MDT0001 complete [11322.509930] XFS (loop0): Unmounting Filesystem [11327.580726] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 22:54:55 (1788663295) [11335.413066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.067619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11336.079515] Lustre: Skipped 2 previous similar messages [11341.592351] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11351.498786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.117463] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11365.133953] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11371.142449] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11378.268681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11384.391548] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11391.061185] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11406.307512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11406.310355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11406.340491] Lustre: Skipped 2 previous similar messages [11406.350289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11406.369216] Lustre: Skipped 4 previous similar messages [11409.891700] Lustre: server umount lustre-OST0000 complete [11426.795799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11426.801211] Lustre: Skipped 5 previous similar messages [11428.319163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11428.459477] Lustre: server umount lustre-MDT0000 complete [11431.910506] LustreError: 347564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11431.946838] LustreError: 347564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11432.798190] LustreError: 347544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788663402 with bad export cookie 13011397922974043391 [11432.805940] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11432.826790] LustreError: 347544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11433.264386] Lustre: server umount lustre-MDT0001 complete [11442.935189] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [11446.672275] Key type lgssc unregistered [11447.119537] LNet: 351196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11447.124053] LNetError: 351196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11447.151921] LNet: Removed LNI 192.168.201.157@tcp [11448.595186] Key type .llcrypt unregistered [11448.598531] Key type ._llcrypt unregistered [11480.561104] Key type ._llcrypt registered [11480.562270] Key type .llcrypt registered [11480.614487] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11481.561685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11481.578133] alg: No test for adler32 (adler32-zlib) [11482.534806] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [11482.916385] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [11484.711489] Key type lgssc registered [11486.292694] Lustre: Echo OBD driver; http://www.lustre.org/ [11493.878756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11493.893035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.441361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11500.190307] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11505.634345] LustreError: 352760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11505.645793] LustreError: 352760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11508.831824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.211745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11513.991941] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11521.127096] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11526.527744] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11534.352631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11534.749123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11543.989412] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11552.178393] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11559.901924] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [11560.421734] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11560.428865] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.440388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11564.976949] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [11564.988798] Lustre: Skipped 1 previous similar message [11565.932370] Lustre: server umount lustre-MDT0001 complete [11570.095381] LustreError: 352755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.220052] LustreError: 354002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.265723] LustreError: 354002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11576.320029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.404536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11576.411221] Lustre: Found index 1 for lustre-MDT0001, updating log [11576.448737] Lustre: 352747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11576.711030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11581.391635] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11581.950428] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11585.562380] LustreError: 356363:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11585.578770] LustreError: 356363:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11585.587184] LustreError: 352751:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11585.684803] LustreError: 356366:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.157@tcp: cfg command failed: rc = -17 [11585.690060] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11585.690060] [11887.012879] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.026811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11887.033153] Lustre: Skipped 2 previous similar messages [11887.048211] Lustre: Skipped 3 previous similar messages [11892.204282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11892.213110] Lustre: Skipped 2 previous similar messages [11900.384387] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11900.669770] Lustre: server umount lustre-MDT0001 complete [11902.434286] LustreError: 354011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.453190] LustreError: 354011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11907.575663] LustreError: 358845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11907.601832] LustreError: 358845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11910.840085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.890666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11910.893996] Lustre: Found index 1 for lustre-MDT0001, updating log [11910.923886] Lustre: 352749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11911.149900] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11915.610631] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11916.279935] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11920.394111] LustreError: 364853:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11920.407713] LustreError: 364853:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11920.432386] LustreError: 352751:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11920.435984] LustreError: 352751:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11920.548449] LustreError: 364863:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11920.552340] LustreError: 364863:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11920.555338] LustreError: 364863:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.157@tcp: cfg command failed: rc = -17 [11920.560094] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.157@tcp [11920.560094] [12221.413176] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.423504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12221.428346] Lustre: Skipped 5 previous similar messages [12227.601462] Lustre: server umount lustre-MDT0001 complete [12230.585703] LustreError: 354002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.629349] LustreError: 354002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12238.464718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.562062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12238.568682] Lustre: Found index 1 for lustre-MDT0001, updating log [12238.605222] Lustre: 352748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12238.924312] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12242.493407] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12243.949616] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12247.043755] LustreError: 373232:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12247.053924] LustreError: 373232:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12247.059178] LustreError: 352751:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12247.063980] LustreError: 352751:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12247.136537] LustreError: 373235:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12247.147333] LustreError: 373235:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.157@tcp: cfg command failed: rc = -114 [12247.154826] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12247.154826] [12566.497651] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.508795] Lustre: Skipped 1 previous similar message [12566.509063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12566.515033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12566.515042] Lustre: Skipped 3 previous similar messages [12570.293727] Lustre: server umount lustre-MDT0000 complete [12571.616618] LustreError: 352756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12571.626917] LustreError: 352756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12573.982700] LustreError: 355949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664543 with bad export cookie 8134201879415643152 [12573.996794] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.002091] LustreError: 355949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12576.738963] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12576.759070] Lustre: Skipped 1 previous similar message [12580.363848] Lustre: server umount lustre-MDT0001 complete [12584.040224] Lustre: server umount lustre-OST0000 complete [12593.612842] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [12603.962816] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12637.693358] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 23:16:45 (1788664605) [12644.002618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12644.225744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12644.257094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12644.313893] Lustre: lustre-MDT0000: new disk, initializing [12644.402561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12644.426385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12648.831917] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12658.284873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12658.425267] Lustre: 384002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12658.498482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12658.502533] Lustre: Skipped 1 previous similar message [12658.610473] Lustre: lustre-MDT0001: new disk, initializing [12658.689743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12658.738604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12658.749449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12663.509175] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12672.250579] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12677.466506] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12683.024280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12683.223669] Lustre: lustre-OST0000: new disk, initializing [12683.226886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12683.232135] Lustre: 385967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12683.287830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12688.615354] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12689.450020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12689.463308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12689.545756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12698.533946] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12707.842865] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12707.856569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12707.866405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12707.872035] Lustre: Skipped 4 previous similar messages [12714.556468] Lustre: server umount lustre-MDT0000 complete [12714.978564] LustreError: 384007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12715.020994] LustreError: 384007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12718.676538] LustreError: 383992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664687 with bad export cookie 8134201879415645105 [12718.685743] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12718.691436] LustreError: 383992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12718.691637] LustreError: 384806:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12719.242251] Lustre: server umount lustre-MDT0001 complete [12733.993688] Lustre: server umount lustre-OST0000 complete [12767.413881] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [12778.503445] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12808.227941] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12500 sec ========= 23:19:35 (1788664775) [12810.193557] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:19:37 (1788664777) === [12814.023351] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:19:41 (1788664781) === [12854.459634] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [12857.168464] Key type lgssc unregistered [12857.533548] LNet: 392335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12857.541199] LNetError: 392335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12857.569143] LNet: Removed LNI 192.168.201.157@tcp [12858.348331] Key type .llcrypt unregistered [12858.351333] Key type ._llcrypt unregistered