[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489826597 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007961] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011061] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.015074] *** VALIDATE selinux *** [ 0.023005] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028667] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032067] *** VALIDATE tmpfs *** [ 0.033450] *** VALIDATE proc *** [ 0.034262] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042878] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.044878] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051010] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053322] rcu: Hierarchical SRCU implementation. [ 0.055457] smp: Bringing up secondary CPUs ... [ 0.056621] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060076] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127365] node 0 deferred pages initialised in 62ms [ 0.130219] devtmpfs: initialized [ 0.131251] x86/mm: Memory block size: 128MB [ 0.134588] gcov: version magic: 0x41383552 [ 0.137171] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142305] pinctrl core: initialized pinctrl subsystem [ 0.144159] [ 0.144833] ************************************************************* [ 0.147013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150011] ** ** [ 0.152011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154012] ** ** [ 0.157014] ** This means that this kernel is built to expose internal ** [ 0.159011] ** IOMMU data structures, which may compromise security on ** [ 0.161016] ** your system. ** [ 0.164011] ** ** [ 0.166010] ** If you see this message and you are not debugging the ** [ 0.168010] ** kernel, report this immediately to your vendor! ** [ 0.170015] ** ** [ 0.173010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175010] ************************************************************* [ 0.178697] NET: Registered protocol family 16 [ 0.179274] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187484] cpuidle: using governor menu [ 0.188746] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190577] PCI: Using configuration type 1 for base access [ 0.192112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206076] cryptd: max_cpu_qlen set to 1000 [ 0.208221] ACPI: Added _OSI(Module Device) [ 0.210016] ACPI: Added _OSI(Processor Device) [ 0.212014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218930] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225545] ACPI: Interpreter enabled [ 0.227066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228028] ACPI: Using IOAPIC for interrupt routing [ 0.230109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233283] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242874] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259364] acpiphp: Slot [2] registered [ 0.261124] acpiphp: Slot [5] registered [ 0.262137] acpiphp: Slot [6] registered [ 0.264126] acpiphp: Slot [7] registered [ 0.266128] acpiphp: Slot [8] registered [ 0.268157] acpiphp: Slot [9] registered [ 0.269132] acpiphp: Slot [10] registered [ 0.271118] acpiphp: Slot [3] registered [ 0.272100] acpiphp: Slot [4] registered [ 0.274310] acpiphp: Slot [11] registered [ 0.276082] acpiphp: Slot [12] registered [ 0.277102] acpiphp: Slot [13] registered [ 0.279143] acpiphp: Slot [14] registered [ 0.281108] acpiphp: Slot [15] registered [ 0.282107] acpiphp: Slot [16] registered [ 0.284108] acpiphp: Slot [17] registered [ 0.285079] acpiphp: Slot [18] registered [ 0.286075] acpiphp: Slot [19] registered [ 0.287088] acpiphp: Slot [20] registered [ 0.288069] acpiphp: Slot [21] registered [ 0.289083] acpiphp: Slot [22] registered [ 0.290075] acpiphp: Slot [23] registered [ 0.291096] acpiphp: Slot [24] registered [ 0.293077] acpiphp: Slot [25] registered [ 0.294079] acpiphp: Slot [26] registered [ 0.295098] acpiphp: Slot [27] registered [ 0.296186] acpiphp: Slot [28] registered [ 0.297065] acpiphp: Slot [29] registered [ 0.298102] acpiphp: Slot [30] registered [ 0.299074] acpiphp: Slot [31] registered [ 0.299959] PCI host bridge to bus 0000:00 [ 0.301014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.302014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.304015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311816] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314025] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328505] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339093] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.341696] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.344042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348100] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.367017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.373007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380607] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424878] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.446014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.473387] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.483017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.488016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509582] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.518013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.525012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.552776] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.558013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.580015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.590259] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.595011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.600013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.615016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.626065] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.629385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.631940] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.634282] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.636255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.640039] iommu: Default domain type: Passthrough [ 0.642655] SCSI subsystem initialized [ 0.644124] ACPI: bus type USB registered [ 0.645087] usbcore: registered new interface driver usbfs [ 0.648087] usbcore: registered new interface driver hub [ 0.650083] usbcore: registered new device driver usb [ 0.651307] pps_core: LinuxPPS API ver. 1 registered [ 0.653011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.656060] PTP clock support registered [ 0.658126] EDAC MC: Ver: 3.0.0 [ 0.660606] PCI: Using ACPI for IRQ routing [ 0.662909] NetLabel: Initializing [ 0.664011] NetLabel: domain hash size = 128 [ 0.665009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.667108] NetLabel: unlabeled traffic allowed by default [ 0.670122] vgaarb: loaded [ 0.671284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.672015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.679411] clocksource: Switched to clocksource kvm-clock [ 0.778695] VFS: Disk quotas dquot_6.6.0 [ 0.780363] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.784350] *** VALIDATE ramfs *** [ 0.785865] *** VALIDATE hugetlbfs *** [ 0.788700] pnp: PnP ACPI init [ 0.791402] pnp: PnP ACPI: found 6 devices [ 0.810345] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.814126] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.816414] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.818739] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.821536] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.824482] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.827740] NET: Registered protocol family 2 [ 0.829959] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.834904] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.838642] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.845394] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.848879] TCP: Hash tables configured (established 65536 bind 65536) [ 0.851771] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.854913] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857796] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860910] NET: Registered protocol family 1 [ 0.863794] RPC: Registered named UNIX socket transport module. [ 0.865654] RPC: Registered udp transport module. [ 0.866783] RPC: Registered tcp transport module. [ 0.868028] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870834] NET: Registered protocol family 44 [ 0.873033] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.874981] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.877432] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.880000] PCI: CLS 0 bytes, default 64 [ 0.881348] Unpacking initramfs... [ 2.321118] debug: unmapping init [mem 0xffff924cbcc54000-0xffff924cbffbffff] [ 2.326079] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.328476] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.331580] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.845425] Initialise system trusted keyrings [ 2.847587] Key type blacklist registered [ 2.849850] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.860295] zbud: loaded [ 2.864879] *** VALIDATE nfs *** [ 2.866656] *** VALIDATE nfs4 *** [ 2.868482] pstore: using deflate compression [ 2.872331] Platform Keyring initialized [ 2.986950] NET: Registered protocol family 38 [ 2.988881] Key type asymmetric registered [ 2.990706] Asymmetric key parser 'x509' registered [ 2.992848] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.996336] io scheduler mq-deadline registered [ 2.998257] io scheduler kyber registered [ 3.000196] io scheduler bfq registered [ 3.002408] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.005949] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.009301] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.012798] ACPI: Power Button [PWRF] [ 3.018945] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.026398] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.050835] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.061580] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.081963] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.112333] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.144175] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.152141] Non-volatile memory driver v1.3 [ 3.153847] Linux agpgart interface v0.103 [ 3.194312] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.197639] vda: detected capacity change from 0 to 74698752 [ 3.215590] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.218924] vdb: detected capacity change from 0 to 1073741824 [ 3.237247] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.240694] vdc: detected capacity change from 0 to 2621440000 [ 3.257462] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.261524] vdd: detected capacity change from 0 to 2621440000 [ 3.279726] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.283682] vde: detected capacity change from 0 to 4294967296 [ 3.301670] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.305299] vdf: detected capacity change from 0 to 4294967296 [ 3.314227] libphy: Fixed MDIO Bus: probed [ 3.319674] usbcore: registered new interface driver usbserial_generic [ 3.322701] usbserial: USB Serial support registered for generic [ 3.325661] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.330911] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.334854] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.337531] mousedev: PS/2 mouse device common for all mice [ 3.340504] rtc_cmos 00:05: RTC can wake from S4 [ 3.343962] rtc_cmos 00:05: registered as rtc0 [ 3.345725] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.348283] intel_pstate: CPU model not supported [ 3.350504] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.351706] hid: raw HID events driver (C) Jiri Kosina [ 3.359235] usbcore: registered new interface driver usbhid [ 3.359576] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.360941] usbhid: USB HID core driver [ 3.361234] drop_monitor: Initializing network drop monitor service [ 3.366870] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.368167] Initializing XFRM netlink socket [ 3.373532] NET: Registered protocol family 10 [ 3.376369] Segment Routing with IPv6 [ 3.377637] NET: Registered protocol family 17 [ 3.379576] mpls_gso: MPLS GSO support [ 3.385068] RAS: Correctable Errors collector initialized. [ 3.387533] AVX version of gcm_enc/dec engaged. [ 3.389506] AES CTR mode by8 optimization enabled [ 3.467114] sched_clock: Marking stable (3467042476, 0)->(4338214681, -871172205) [ 3.471394] registered taskstats version 1 [ 3.473448] Loading compiled-in X.509 certificates [ 3.475751] zswap: loaded using pool lzo/zbud [ 3.504542] Key type big_key registered [ 3.516771] Key type encrypted registered [ 3.518838] ima: No TPM chip found, activating TPM-bypass! [ 3.520937] ima: Allocated hash algorithm: sha1 [ 3.523215] ima: No architecture policies found [ 3.525099] evm: Initialising EVM extended attributes: [ 3.527340] evm: security.selinux [ 3.528760] evm: security.ima [ 3.529849] evm: security.capability [ 3.531399] evm: HMAC attrs: 0x1 [ 3.534483] rtc_cmos 00:05: setting system clock to 2026-09-03 14:02:42 UTC (1788444162) [ 3.541851] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 3.545352] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 3.554131] Write protecting the kernel read-only data: 28672k [ 3.557594] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.560249] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 3.595362] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.605602] systemd[1]: Detected virtualization kvm. [ 3.607444] systemd[1]: Detected architecture x86-64. [ 3.609854] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.634139] systemd[1]: No hostname configured. [ 3.636416] systemd[1]: Set hostname to . [ 3.638365] random: systemd: uninitialized urandom read (16 bytes read) [ 3.640899] systemd[1]: Initializing machine ID from random generator. [ 3.702187] random: ln: uninitialized urandom read (6 bytes read) [ 3.794943] random: systemd: uninitialized urandom read (16 bytes read) [ 3.798143] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.803855] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.812443] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.505268] device-mapper: uevent: version 1.0.3 [ 4.508134] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.286380] random: fast init done [ 5.291908] virtio_net virtio0 ens2: renamed from eth0 [ 5.308488] scsi host0: ata_piix [ 5.324187] scsi host1: ata_piix [ 5.325734] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.328483] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.041824] random: crng init done [ 10.044774] random: 7 urandom warning(s) missed due to ratelimiting [ 10.066368] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.021098] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.048623] printk: systemd: 24 output lines suppressed due to ratelimiting [ 14.960777] SELinux: Disabled at runtime. [ 15.044923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.058576] systemd[1]: Detected virtualization kvm. [ 15.064883] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.595091] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.598633] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.604978] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.609234] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.612539] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.624089] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.651875] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ 16.937628] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.257158] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.141062] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.262786] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.717230] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.855171] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.077178] Key type dns_resolver registered [ 26.212321] hrtimer: interrupt took 2620296 ns [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 27.263224] NFS: Registering the id_resolver key type [ 27.275252] Key type id_resolver registered [ 27.283489] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg314-server login: [ 91.495224] libcfs: loading out-of-tree module taints kernel. [ 91.598362] Key type ._llcrypt registered [ 91.602077] Key type .llcrypt registered [ 91.713518] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 110.954572] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 112.761328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.773307] alg: No test for adler32 (adler32-zlib) [ 114.290569] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 115.108245] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 116.968197] Key type lgssc registered [ 119.193455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.344406] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 181.295447] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 197.960519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.015571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.362839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.428375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.606805] Lustre: lustre-MDT0000: new disk, initializing [ 199.761104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.807260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.859684] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.651082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.770570] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.812876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.818421] Lustre: Skipped 1 previous similar message [ 219.925804] Lustre: lustre-MDT0001: new disk, initializing [ 220.038826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.079233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.087084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 225.555215] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.041505] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 242.591793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.077512] Lustre: lustre-OST0000: new disk, initializing [ 243.085673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.095311] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.217755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.773210] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.972975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.988845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.079443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.065726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.244609] Lustre: lustre-OST0001: new disk, initializing [ 265.247950] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 265.253075] Lustre: 9527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.330937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.694885] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.053687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 274.066671] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 274.143454] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.778557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 291.537365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 298.923552] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing check_logdir /tmp/testlogs/ [ 305.056741] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing yml_node [ 310.288474] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 312.977166] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 315.905332] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 317.732927] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 10:07:54 EDT 2026 [ 338.284910] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 340.485306] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 358.370824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 358.391685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.446459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.441537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.444173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.086170] Lustre: server umount lustre-MDT0000 complete [ 368.621441] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444527 with bad export cookie 8252423214740856686 [ 368.622935] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.648244] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 371.682206] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.682883] LustreError: 9534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.683371] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.683377] Lustre: Skipped 3 previous similar messages [ 371.695113] Lustre: Skipped 3 previous similar messages [ 371.740520] LustreError: 9534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 373.729480] LustreError: 12270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.729606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.756038] LustreError: 12270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 373.775110] Lustre: Skipped 2 previous similar messages [ 375.121486] Lustre: server umount lustre-MDT0001 complete [ 379.446533] Lustre: server umount lustre-OST0000 complete [ 383.313712] Lustre: server umount lustre-OST0001 complete [ 389.703055] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 397.827816] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 441.267861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.459668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.493282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.607704] Lustre: lustre-MDT0000: new disk, initializing [ 441.717975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.767570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.031491] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 458.185509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.374618] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 458.466345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 458.478235] Lustre: Skipped 1 previous similar message [ 458.633595] Lustre: lustre-MDT0001: new disk, initializing [ 458.805694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.859758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.872101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 463.398252] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 475.460893] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.881875] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.129472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.488450] Lustre: lustre-OST0000: new disk, initializing [ 488.498846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.507316] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.637582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.196709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.208395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.333954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 494.523775] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 504.895215] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 509.330449] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 509.624078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 514.145293] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 514.412561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.195290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.199078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.203542] LustreError: Skipped 1 previous similar message [ 521.205162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.228489] Lustre: Skipped 1 previous similar message [ 523.065522] Lustre: server umount lustre-OST0000 complete [ 530.401685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.411336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.434016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.441127] Lustre: Skipped 1 previous similar message [ 541.152266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 541.382602] Lustre: server umount lustre-MDT0000 complete [ 541.666506] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.687496] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 545.225418] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444704 with bad export cookie 8252423214740858226 [ 545.234669] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.239528] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 545.654986] Lustre: server umount lustre-MDT0001 complete [ 550.378340] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:11:47 (1788444707) [ 556.353514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.077646] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.163249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.726990] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 562.682853] LustreError: 19320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.387693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.987989] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 586.221684] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 592.701498] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 601.924940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 609.845266] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 618.494360] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 622.372336] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 632.291238] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.302803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.317472] Lustre: Skipped 3 previous similar messages [ 632.339568] Lustre: Skipped 6 previous similar messages [ 636.129773] Lustre: server umount lustre-MDT0000 complete [ 637.422115] LustreError: 20765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.464257] LustreError: 20765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 641.420453] LustreError: 19300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444800 with bad export cookie 8252423214740859101 [ 641.431431] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.433835] LustreError: 19300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 641.999726] Lustre: server umount lustre-MDT0001 complete [ 657.149668] Lustre: server umount lustre-OST0000 complete [ 678.525899] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 689.206712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.684104] LustreError: 23703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.750888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.759385] Lustre: Skipped 2 previous similar messages [ 694.137848] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 702.848476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.967716] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 710.825373] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 717.476991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.638959] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 734.969784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.191596] Lustre: lustre-OST0001: new disk, initializing [ 735.199782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 735.210097] Lustre: 26018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 736.431416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 736.447399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 736.559272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 742.171911] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 751.572291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 755.222130] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 767.468339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.470118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.496168] Lustre: Skipped 3 previous similar messages [ 767.512596] Lustre: Skipped 3 previous similar messages [ 772.352398] Lustre: server umount lustre-MDT0000 complete [ 772.603355] LustreError: 25022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.614172] LustreError: 25022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 776.688954] LustreError: 23686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444935 with bad export cookie 8252423214740859955 [ 776.691654] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.699421] LustreError: 23686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 777.212541] Lustre: server umount lustre-MDT0001 complete [ 791.784225] Lustre: server umount lustre-OST0000 complete [ 793.896733] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444936/real 1788444936] req@ffff924c08db3100 x1875319743302400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788444952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 797.145275] Lustre: server umount lustre-OST0001 complete [ 815.416901] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 826.756237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.429670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.448666] Lustre: Skipped 3 previous similar messages [ 833.082641] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 837.602055] LustreError: 28987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 837.634250] LustreError: 28987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 842.737287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.020726] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 851.161624] Lustre: 30137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 857.544504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 864.938288] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 874.057676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.139560] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 892.441849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 897.265717] Lustre: 32013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 904.676649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 904.701831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.752107] Lustre: Skipped 1 previous similar message [ 904.756156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.862829] Lustre: server umount lustre-MDT0000 complete [ 912.808335] LustreError: 28969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445071 with bad export cookie 8252423214740861250 [ 912.828635] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.831510] LustreError: 28969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 913.527606] Lustre: server umount lustre-MDT0001 complete [ 927.720622] Lustre: server umount lustre-OST0000 complete [ 941.923830] Lustre: server umount lustre-OST0001 complete [ 952.236898] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:18:28 (1788445108) [ 984.598501] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 995.834783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.389971] LustreError: 34820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.412827] LustreError: 34820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 1002.323682] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1011.104914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.418861] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1018.988513] Lustre: 35969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1026.482133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.837301] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1043.358384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.606433] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1056.743736] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1060.196442] Lustre: 37849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1068.001429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1068.006519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.033897] Lustre: Skipped 3 previous similar messages [ 1068.048811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.072426] Lustre: Skipped 4 previous similar messages [ 1071.263969] Lustre: server umount lustre-MDT0000 complete [ 1075.204247] LustreError: 34801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445234 with bad export cookie 8252423214740862356 [ 1075.206132] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.214136] LustreError: 34801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1075.761088] Lustre: server umount lustre-MDT0001 complete [ 1090.997275] Lustre: server umount lustre-OST0000 complete [ 1104.882860] Lustre: server umount lustre-OST0001 complete [ 1113.999486] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:21:10 (1788445270) [ 1137.233861] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1150.359668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.045572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.059384] Lustre: Skipped 7 previous similar messages [ 1156.460153] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1164.644135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.581443] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1172.782625] Lustre: 41161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1179.472705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.293833] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1195.394637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.698369] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1212.451429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1217.633636] Lustre: 43039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1218.552114] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1218.557301] Lustre: Skipped 1 previous similar message [ 1221.602078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1221.613982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.637226] Lustre: Skipped 3 previous similar messages [ 1227.950796] Lustre: server umount lustre-MDT0000 complete [ 1231.280462] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445390 with bad export cookie 8252423214740863462 [ 1231.297570] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.308770] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1237.832307] Lustre: server umount lustre-MDT0001 complete [ 1245.935256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.929835] LustreError: 3648:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff924d08804380 x1875319743455744/t0(0) o250->MGC192.168.203.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1257.205026] LustreError: 41524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.221143] LustreError: 41524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 1263.234666] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1266.614348] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1273.413217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.102683] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1279.496968] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1279.509394] Lustre: Skipped 1 previous similar message [ 1287.109481] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1288.874604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1294.098601] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1295.587294] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1305.073287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1307.444322] Lustre: server umount lustre-MDT0000 complete [ 1311.588840] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445470 with bad export cookie 8252423214740864589 [ 1311.595217] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.611943] LustreError: 39995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1311.793449] Lustre: server umount lustre-MDT0001 complete [ 1326.680437] Lustre: server umount lustre-OST0000 complete [ 1341.973137] Lustre: server umount lustre-OST0001 complete [ 1350.608460] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:25:07 (1788445507) [ 1366.286522] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1375.547606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.694249] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1389.495871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.703512] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1397.615472] Lustre: 48803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1404.673896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.672366] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1422.610419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.290855] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1444.832123] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1451.009420] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1464.297121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.303980] Lustre: Skipped 20 previous similar messages [ 1466.525928] Lustre: server umount lustre-MDT0000 complete [ 1470.268308] LustreError: 47631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445629 with bad export cookie 8252423214740865387 [ 1470.272621] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1470.283939] LustreError: 47631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1470.659580] Lustre: server umount lustre-MDT0001 complete [ 1485.036977] Lustre: server umount lustre-OST0000 complete [ 1499.126569] Lustre: server umount lustre-OST0001 complete [ 1509.681920] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:27:46 (1788445666) [ 1512.110696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1514.155275] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:27:50 (1788445670) [ 1526.139927] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 1534.558295] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1558.271811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.686929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1558.801971] Lustre: lustre-MDT0000: new disk, initializing [ 1558.983236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1565.067233] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1574.520240] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1581.561433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.893591] Lustre: lustre-OST0000: new disk, initializing [ 1581.899809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1581.906323] Lustre: Skipped 1 previous similar message [ 1581.922288] Lustre: 54878:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1583.801628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1583.815436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1583.868942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1588.088359] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1597.482065] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1599.758054] Lustre: DEBUG MARKER: create test files [ 1635.298049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.312540] Lustre: Skipped 13 previous similar messages [ 1638.796298] Lustre: server umount lustre-OST0000 complete [ 1642.900483] Lustre: server umount lustre-MDT0000 complete [ 1654.889461] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1657.749439] Key type lgssc unregistered [ 1658.066034] LNet: 56511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.077514] LNetError: 56511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.103581] LNet: Removed LNI 192.168.203.114@tcp [ 1659.278162] Key type .llcrypt unregistered [ 1659.299951] Key type ._llcrypt unregistered [ 1688.938608] Key type ._llcrypt registered [ 1688.941811] Key type .llcrypt registered [ 1689.081814] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1690.423732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1690.470046] alg: No test for adler32 (adler32-zlib) [ 1691.555682] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1691.840770] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1693.544189] Key type lgssc registered [ 1694.670956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1702.888685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1702.919819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.549610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.427966] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1717.295416] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.728666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.427163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.519204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1729.656206] Lustre: 59245:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1729.664946] Lustre: 59245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1729.670578] Lustre: 59245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1729.679407] Lustre: 59245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1729.694926] Lustre: 59245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1729.702477] Lustre: 59245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1731.840170] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1739.200391] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.675794] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1750.414076] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1760.230715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1760.238331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.245729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1764.865543] Lustre: server umount lustre-OST0000 complete [ 1769.605213] Lustre: server umount lustre-MDT0000 complete [ 1781.460988] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1784.543817] Key type lgssc unregistered [ 1784.921673] LNet: 60861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.927879] LNetError: 60861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.955225] LNet: Removed LNI 192.168.203.114@tcp [ 1785.756246] Key type .llcrypt unregistered [ 1785.757978] Key type ._llcrypt unregistered [ 1814.791504] Key type ._llcrypt registered [ 1814.794899] Key type .llcrypt registered [ 1814.894129] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1815.489816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1815.524819] alg: No test for adler32 (adler32-zlib) [ 1816.661442] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1816.841694] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1818.504294] Key type lgssc registered [ 1819.738149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.273229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.321404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.816407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.007494] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1840.818301] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1849.462932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.932494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.021576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1854.229902] Lustre: 63696:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1854.243071] Lustre: 63696:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1854.257032] Lustre: 63696:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1854.269148] Lustre: 63696:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1854.280887] Lustre: 63696:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1854.287809] Lustre: 63696:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1857.974556] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1867.066648] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1873.000916] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1884.648329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.650066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1884.680566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1888.535887] Lustre: server umount lustre-OST0000 complete [ 1893.877674] Lustre: server umount lustre-MDT0000 complete [ 1907.659597] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1911.118090] Key type lgssc unregistered [ 1911.454616] LNet: 65259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.471809] LNetError: 65259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.504063] LNet: Removed LNI 192.168.203.114@tcp [ 1912.390143] Key type .llcrypt unregistered [ 1912.392158] Key type ._llcrypt unregistered [ 1934.576914] Key type ._llcrypt registered [ 1934.582043] Key type .llcrypt registered [ 1934.747135] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 1951.857929] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1953.390132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.515262] alg: No test for adler32 (adler32-zlib) [ 1954.569691] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1954.832121] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1956.544373] Key type lgssc registered [ 1957.820869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1994.234970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1994.272623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.603861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1995.657174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1995.756721] Lustre: lustre-MDT0000: new disk, initializing [ 1995.843631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.874681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2001.038453] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2013.453209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.613327] Lustre: 68412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2013.699287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2013.707112] Lustre: Skipped 1 previous similar message [ 2013.832253] Lustre: lustre-MDT0001: new disk, initializing [ 2013.991035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.051256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2014.069515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2019.218268] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2029.739958] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2036.187485] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.514635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2044.045697] Lustre: lustre-OST0000: new disk, initializing [ 2044.052580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2044.068547] Lustre: 70383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2044.241646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.236432] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2054.322084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2054.329868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2054.569040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2061.127319] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2066.420362] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2066.672384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2071.680698] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2071.903323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2075.106600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.116609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.127208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2075.143172] Lustre: Skipped 1 previous similar message [ 2080.234975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2080.242424] Lustre: Skipped 1 previous similar message [ 2080.496924] Lustre: server umount lustre-OST0000 complete [ 2085.858851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2085.870742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.882611] Lustre: Skipped 1 previous similar message [ 2085.895261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.468581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.469664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.517273] Lustre: Skipped 1 previous similar message [ 2092.028080] Lustre: server umount lustre-MDT0000 complete [ 2095.797090] LustreError: 68406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446254 with bad export cookie 9471731127203098763 [ 2095.799714] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.808855] LustreError: 68406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2096.232525] Lustre: server umount lustre-MDT0001 complete [ 2105.351540] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:37:42 (1788446262) [ 2115.939815] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2153.817905] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 2163.013947] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2195.565891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.813797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2195.862707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2195.976299] Lustre: lustre-MDT0000: new disk, initializing [ 2196.035905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.047247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2199.790561] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2209.836227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.046952] Lustre: 76404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2210.158483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2210.166909] Lustre: Skipped 1 previous similar message [ 2210.261371] Lustre: lustre-MDT0001: new disk, initializing [ 2210.358973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2210.400900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2210.423064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2215.368338] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2225.874654] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2232.242907] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2240.117413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.318537] Lustre: lustre-OST0000: new disk, initializing [ 2240.321433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2240.326060] Lustre: 78372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2240.394349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.480961] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2248.247503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2248.256799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2248.382547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2253.557842] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2257.657650] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2257.869331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2261.590549] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2261.796792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2264.043246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.048445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.055971] LustreError: Skipped 1 previous similar message [ 2264.058992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.068547] Lustre: Skipped 1 previous similar message [ 2264.089434] Lustre: Skipped 1 previous similar message [ 2269.614669] Lustre: server umount lustre-OST0000 complete [ 2274.276575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.283931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.306350] Lustre: Skipped 1 previous similar message [ 2274.324974] Lustre: Skipped 3 previous similar messages [ 2279.491141] Lustre: server umount lustre-MDT0000 complete [ 2282.916241] LustreError: 76394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446441 with bad export cookie 9471731127203099645 [ 2282.924983] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.929880] LustreError: 76394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2283.447122] Lustre: server umount lustre-MDT0001 complete [ 2291.800547] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:40:48 (1788446448) [ 2298.483413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.773425] LustreError: 80784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.787510] LustreError: 80784:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2298.842464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.755041] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2303.975307] LustreError: 80785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.070029] LustreError: 80784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.976741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.091847] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2323.166138] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2328.115327] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2333.734481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2334.005170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.020042] Lustre: Skipped 1 previous similar message [ 2340.137431] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2347.361962] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2350.577815] Lustre: 83455:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 678ddb17-a8a1-4172-bb37-f58b97598d7f at adminstrative request [ 2353.642395] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.669727] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2363.887914] Lustre: MGS: Client 678ddb17-a8a1-4172-bb37-f58b97598d7f (at 0@lo) reconnecting [ 2363.901501] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x83725946dcf3776f to 0x83725946dcf37a0f [ 2363.924975] Lustre: MGC192.168.203.114@tcp: Connection restored to 0@lo (at 0@lo) [ 2387.585103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.760950] Lustre: lustre-OST0001: new disk, initializing [ 2387.764607] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2387.770183] Lustre: 83848:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2393.215628] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2397.798153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2397.815435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2397.920487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2402.494320] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2408.417948] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2408.424812] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.430358] LustreError: Skipped 1 previous similar message [ 2408.431368] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2408.452851] Lustre: Skipped 1 previous similar message [ 2408.453634] Lustre: Skipped 2 previous similar messages [ 2411.517296] Lustre: server umount lustre-OST0001 complete [ 2413.546883] LustreError: 82607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.574396] LustreError: 82607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2418.666195] LustreError: 82604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.691918] LustreError: 82604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2421.762249] Lustre: server umount lustre-OST0000 complete [ 2428.903244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.927994] Lustre: Skipped 3 previous similar messages [ 2431.531712] Lustre: server umount lustre-MDT0000 complete [ 2434.019655] LustreError: 80780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.032874] LustreError: 80780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2434.902826] LustreError: 80766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446593 with bad export cookie 9471731127203101199 [ 2434.913203] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.922897] LustreError: 80766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2435.426972] Lustre: server umount lustre-MDT0001 complete [ 2443.574851] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:43:20 (1788446600) [ 2445.181354] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2447.106282] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:43:24 (1788446604) [ 2448.654922] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2450.496460] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:43:27 (1788446607) [ 2451.809798] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2453.461224] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 10:43:30 (1788446610) [ 2454.978564] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2456.371525] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:43:33 (1788446613) [ 2457.935725] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2463.384290] loop: module loaded [ 2463.403844] loop0: detected capacity change from 0 to 4294967296 [ 2463.455932] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2464.871567] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2468.098968] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2471.216957] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2494.909699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.934763] LustreError: 87970:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2494.949315] LustreError: 87970:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2494.955108] LustreError: 87970:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2494.960791] LustreError: 87970:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2511.332537] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 2522.147807] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2559.089250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.264966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2559.295068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2559.379127] Lustre: lustre-MDT0000: new disk, initializing [ 2559.493133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.500771] Lustre: Skipped 1 previous similar message [ 2559.521388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2564.514553] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2573.841394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.965871] Lustre: 90837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2574.175143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2578.163827] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2586.540708] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2591.133631] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.955606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.244334] Lustre: 92804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2603.254342] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2604.637426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2604.793599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2611.715200] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2615.862547] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2616.131267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2620.377328] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2620.602757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2625.510023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2625.522386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.537189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2625.539886] Lustre: Skipped 4 previous similar messages [ 2628.405470] Lustre: server umount lustre-OST0000 complete [ 2637.819690] Lustre: server umount lustre-MDT0000 complete [ 2640.867601] LustreError: 90847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.881526] LustreError: 90847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2640.991991] LustreError: 92815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446799 with bad export cookie 9471731127203101815 [ 2641.002583] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.005704] LustreError: 92815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2641.274797] Lustre: server umount lustre-MDT0001 complete [ 2649.070927] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:46:46 (1788446806) [ 2655.365528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.146747] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2668.212710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.224867] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2681.216856] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.856774] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2693.903930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.076753] LustreError: 97036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.105952] LustreError: 97036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2694.245238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.251550] Lustre: Skipped 4 previous similar messages [ 2700.877924] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2708.301608] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2713.237841] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2713.373061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2717.061186] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2717.217827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2724.399441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2724.633891] Lustre: lustre-OST0001: new disk, initializing [ 2724.637878] Lustre: Skipped 2 previous similar messages [ 2724.643133] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2724.650792] Lustre: Skipped 3 previous similar messages [ 2724.655926] Lustre: 98585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2732.212476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2732.223022] Lustre: Skipped 2 previous similar messages [ 2732.234229] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2732.333513] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2732.378108] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2741.661937] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2746.318631] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2746.515734] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2750.946394] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2751.193517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2768.261521] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2789.993793] Lustre: Failing over lustre-MDT0000 [ 2790.403615] Lustre: server umount lustre-MDT0000 complete [ 2791.398466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2791.417110] LustreError: Skipped 1 previous similar message [ 2791.426583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.438984] Lustre: Skipped 3 previous similar messages [ 2791.453785] LustreError: 95216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.337044] Lustre: 66636:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446952/real 1788446952] req@000000006a8378e7 x1875321671649408/t0(0) o400->MGC192.168.203.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788446968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2810.382205] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.671058] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2815.675900] LDISKFS-fs (dm-0): recovery complete [ 2815.690699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.579533] LustreError: 66633:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b24c92b7 x1875321671657728/t0(0) o250->MGC192.168.203.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2820.756389] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (not set up) [ 2820.769064] Lustre: Skipped 3 previous similar messages [ 2820.920334] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2825.817854] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2825.876483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2825.877651] Lustre: 95210:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2826.219313] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2826.275272] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2826.601127] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2826.607681] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2826.624640] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 2826.967217] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2826.986510] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2827.312194] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2827.322668] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2827.661111] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2827.676098] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 2 previous similar messages [ 2828.001526] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2828.017644] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2828.344681] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2828.351685] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2829.944914] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2829.964660] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 6 previous similar messages [ 2830.300825] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2830.316658] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 2830.640577] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2830.650328] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 2834.132203] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2834.154207] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 2834.499977] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2834.525285] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2834.848572] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2834.868104] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2842.157237] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2842.186669] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 2842.524693] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2842.534947] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2843.160361] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2843.179790] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 25 previous similar messages [ 2858.413875] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2858.423750] Lustre: 101435:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 50 previous similar messages [ 2858.747243] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2858.761343] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 2859.400874] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2859.413903] LustreError: 101435:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 2885.321131] Lustre: 101435:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2885.339580] Lustre: 101435:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2885.349606] Lustre: 101435:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 040a8255-6d1b-4c6b-9b83-74d8efa45b1b@192.168.203.14@tcp [ 2885.361607] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2885.367278] Lustre: 101435:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@000000000d1286ec x1875321653630080/t0(8589934782) o35->040a8255-6d1b-4c6b-9b83-74d8efa45b1b@192.168.203.14@tcp:34/0 lens 392/0 e 0 to 0 dl 1788447054 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2885.400771] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2885.421127] Lustre: lustre-MDT0000: Denying connection for new client 040a8255-6d1b-4c6b-9b83-74d8efa45b1b (at 192.168.203.14@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2885.432119] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2885.477105] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2885.480018] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2885.504079] Lustre: Skipped 2 previous similar messages [ 2885.520152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:81 to 0x280000401:577) [ 2885.527098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2898.917639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.524711] Lustre: server umount lustre-OST0000 complete [ 2914.602098] Lustre: server umount lustre-OST0001 complete [ 2932.192294] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2932.461550] Lustre: server umount lustre-MDT0000 complete [ 2933.729969] LustreError: 95943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.744710] LustreError: 95943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 44 previous similar messages [ 2936.363582] LustreError: 95196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447095 with bad export cookie 9471731127203153027 [ 2936.365346] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.374447] LustreError: 95196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2936.670271] Lustre: server umount lustre-MDT0001 complete [ 2943.742558] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:51:40 (1788447100) [ 2950.718925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2950.724610] Lustre: *** cfs_fail_loc=197, val=0*** [ 2966.304805] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2971.424811] LustreError: 103414:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 2976.579151] Lustre: *** cfs_fail_loc=197, val=0*** [ 2976.582245] Lustre: Skipped 65 previous similar messages [ 2997.152317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.162252] Lustre: Skipped 2 previous similar messages [ 3004.495090] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3011.941970] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3014.534347] Lustre: server umount lustre-OST0000 complete [ 3021.964384] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:52:58 (1788447178) [ 3043.253162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.100416] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3060.817714] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3065.409118] LustreError: 105067:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3065.420616] LustreError: 105067:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3065.432502] LustreError: 105067:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3065.967230] Lustre: server umount lustre-MDT0000 complete [ 3098.842460] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3108.371625] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3140.586601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.861476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3140.885272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3141.006462] Lustre: lustre-MDT0000: new disk, initializing [ 3141.143329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3145.794597] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3155.516577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.597806] Lustre: 109241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3155.876603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3160.104706] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3168.810044] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3174.056937] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3180.667470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.932559] Lustre: 111211:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3187.109148] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3188.530786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3198.302308] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3204.469976] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3204.768127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3210.748439] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3211.107724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3214.318295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.320273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3214.323758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3214.323767] Lustre: Skipped 14 previous similar messages [ 3214.334778] Lustre: Skipped 10 previous similar messages [ 3214.365820] LustreError: Skipped 4 previous similar messages [ 3219.211234] Lustre: server umount lustre-OST0000 complete [ 3228.898353] Lustre: server umount lustre-MDT0000 complete [ 3229.672344] LustreError: 109255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.708099] LustreError: 109255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3232.404913] LustreError: 110039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447391 with bad export cookie 9471731127203163751 [ 3232.413233] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.419035] LustreError: 110039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3232.757569] Lustre: server umount lustre-MDT0001 complete [ 3240.543230] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:56:37 (1788447397) [ 3249.058540] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3251.595942] Key type lgssc unregistered [ 3251.906080] LNet: 113702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.919856] LNetError: 113702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.958838] LNet: Removed LNI 192.168.203.114@tcp [ 3252.898494] Key type .llcrypt unregistered [ 3252.900943] Key type ._llcrypt unregistered [ 3269.174990] Key type ._llcrypt registered [ 3269.178867] Key type .llcrypt registered [ 3269.293183] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3282.788408] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3283.749837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3283.801087] alg: No test for adler32 (adler32-zlib) [ 3284.893665] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3285.070359] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3286.712154] Key type lgssc registered [ 3288.122405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3330.187856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3330.208152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.390831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3331.414125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3331.467043] Lustre: lustre-MDT0000: new disk, initializing [ 3331.651840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.678056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3335.416115] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3346.452407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3346.835277] Lustre: lustre-OST0000: new disk, initializing [ 3346.838652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3346.843494] Lustre: Skipped 1 previous similar message [ 3346.847769] Lustre: 117872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3346.902594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.702686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3352.715346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3352.763105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3354.012434] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3365.199713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.547262] Lustre: lustre-OST0001: new disk, initializing [ 3365.565705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3365.576992] Lustre: 118735:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3365.683429] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3371.767238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3371.794200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3371.849897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3373.107647] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3382.247777] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.251774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.266703] Lustre: Skipped 1 previous similar message [ 3387.345920] Lustre: server umount lustre-MDT0000 complete [ 3393.998621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3398.151103] LustreError: 117058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447557 with bad export cookie 6545699184470951639 [ 3398.170052] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.176332] LustreError: 117058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3398.269618] Lustre: server umount lustre-OST0000 complete [ 3401.720515] Lustre: server umount lustre-OST0001 complete [ 3410.490431] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:59:27 (1788447567) [ 3432.131981] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:59:49 (1788447589) [ 3443.126847] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3464.365661] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 11:00:21 (1788447621) [ 3502.668951] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3512.863727] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3549.909688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.174755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3550.201046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3550.246565] Lustre: lustre-MDT0000: new disk, initializing [ 3550.321786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.344529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3554.009211] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3564.309659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.449575] Lustre: 126499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3564.505289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3564.508863] Lustre: Skipped 1 previous similar message [ 3564.580965] Lustre: lustre-MDT0001: new disk, initializing [ 3564.669757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.701588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3564.716373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3569.014108] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3578.162554] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3583.339155] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3589.631879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3589.887111] Lustre: lustre-OST0000: new disk, initializing [ 3589.896623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3589.908577] Lustre: 128466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3590.014375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.384961] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3599.968941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3599.973934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3600.130815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3603.399833] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3607.327936] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3607.565328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3612.018363] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3612.358075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3615.715232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.725872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.734880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.741330] Lustre: Skipped 2 previous similar messages [ 3621.140925] Lustre: server umount lustre-OST0000 complete [ 3625.964872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.978151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3625.990358] Lustre: Skipped 1 previous similar message [ 3625.994931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3626.046460] LustreError: Skipped 1 previous similar message [ 3631.081134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3631.096048] Lustre: Skipped 2 previous similar messages [ 3631.399198] Lustre: server umount lustre-MDT0000 complete [ 3635.904689] LustreError: 126491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447794 with bad export cookie 6545699184470952864 [ 3635.906907] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3635.924403] LustreError: 126491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3636.251339] Lustre: server umount lustre-MDT0001 complete [ 3643.302342] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 11:03:20 (1788447800) [ 3649.342776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.678944] LustreError: 130875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.696277] LustreError: 130875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3649.749687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.516323] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3655.141348] LustreError: 130876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.234848] LustreError: 130875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.378584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.102820] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3669.460958] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3674.550269] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3682.035757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3682.458356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.463599] Lustre: Skipped 1 previous similar message [ 3688.745781] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3696.791751] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3701.377430] Lustre: *** cfs_fail_loc=159, val=0*** [ 3716.781041] Lustre: lustre-MDT0000: Client 4774b8f2-fd3d-4da6-a83f-e4cc6bfdee9a (at 192.168.203.14@tcp) reconnecting [ 3718.525536] Lustre: *** cfs_fail_loc=159, val=0*** [ 3718.531033] Lustre: Skipped 7 previous similar messages [ 3734.781852] Lustre: lustre-MDT0000: Client 4774b8f2-fd3d-4da6-a83f-e4cc6bfdee9a (at 192.168.203.14@tcp) reconnecting [ 3752.417762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.422944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.437071] Lustre: Skipped 1 previous similar message [ 3752.446271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.519562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.521340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3758.376690] Lustre: server umount lustre-OST0000 complete [ 3761.636950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3761.643928] Lustre: Skipped 2 previous similar messages [ 3767.927932] Lustre: server umount lustre-MDT0000 complete [ 3771.661372] LustreError: 130857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447930 with bad export cookie 6545699184470953746 [ 3771.678652] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.689614] LustreError: 130857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3772.040291] Lustre: server umount lustre-MDT0001 complete [ 3782.085496] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 3784.450807] Key type lgssc unregistered [ 3784.743886] LNet: 134653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3784.753292] LNetError: 134653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3784.771963] LNet: Removed LNI 192.168.203.114@tcp [ 3785.494105] Key type .llcrypt unregistered [ 3785.499735] Key type ._llcrypt unregistered [ 3812.731749] Key type ._llcrypt registered [ 3812.736871] Key type .llcrypt registered [ 3812.872401] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3813.924992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3814.045531] alg: No test for adler32 (adler32-zlib) [ 3815.068287] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3815.447886] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 3817.123875] Key type lgssc registered [ 3818.356376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3826.337795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3826.385464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.951307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.988068] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3837.415397] LustreError: 136072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.447048] LustreError: 136072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3842.539533] LustreError: 136071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.290649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.933085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3851.143520] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3858.862363] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3865.826750] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3871.848885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.164243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.196733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3877.942469] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3884.983519] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3897.007632] Lustre: *** cfs_fail_loc=159, val=0*** [ 3913.393558] Lustre: lustre-MDT0000: Client 34608dd8-d305-4552-8203-9ba6e66ec5f6 (at 192.168.203.14@tcp) reconnecting [ 3915.198673] Lustre: *** cfs_fail_loc=159, val=0*** [ 3915.207523] Lustre: Skipped 1 previous similar message [ 3930.852064] Lustre: lustre-MDT0001: Client 34608dd8-d305-4552-8203-9ba6e66ec5f6 (at 192.168.203.14@tcp) reconnecting [ 3932.206909] Lustre: *** cfs_fail_loc=159, val=0*** [ 3932.209821] Lustre: Skipped 1 previous similar message [ 3947.784906] Lustre: lustre-MDT0001: Client 34608dd8-d305-4552-8203-9ba6e66ec5f6 (at 192.168.203.14@tcp) reconnecting [ 3953.569993] Lustre: 139114:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3956.334303] Lustre: *** cfs_fail_loc=159, val=0*** [ 3956.343604] Lustre: Skipped 6 previous similar messages [ 3971.792712] Lustre: lustre-MDT0000: Client d26859de-df4f-40d6-8ef3-7443067d935a (at 192.168.203.14@tcp) reconnecting [ 3973.966157] Lustre: *** cfs_fail_loc=159, val=0*** [ 3973.972807] Lustre: Skipped 12 previous similar messages [ 3989.738838] Lustre: lustre-MDT0000: Client d26859de-df4f-40d6-8ef3-7443067d935a (at 192.168.203.14@tcp) reconnecting [ 3991.836462] Lustre: 139414:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3998.185182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3998.191796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3998.210980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.745369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4000.745937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.752680] LustreError: Skipped 8 previous similar messages [ 4000.778266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4003.371302] Lustre: server umount lustre-OST0000 complete [ 4007.906053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4007.913542] LustreError: Skipped 1 previous similar message [ 4007.917508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.927983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.932404] Lustre: Skipped 2 previous similar messages [ 4011.489914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4011.490867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.624040] Lustre: server umount lustre-MDT0000 complete [ 4016.610092] LustreError: 139171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.630415] LustreError: 139171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4017.461365] LustreError: 136052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448176 with bad export cookie 14036891904228740634 [ 4017.482924] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4017.484865] LustreError: 136052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4018.275762] Lustre: server umount lustre-MDT0001 complete [ 4033.295794] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4036.537403] Key type lgssc unregistered [ 4036.856947] LNet: 140475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4036.870729] LNetError: 140475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4036.893503] LNet: Removed LNI 192.168.203.114@tcp [ 4037.847694] Key type .llcrypt unregistered [ 4037.850230] Key type ._llcrypt unregistered [ 4062.414225] Key type ._llcrypt registered [ 4062.416160] Key type .llcrypt registered [ 4062.531765] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4063.464213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4063.496112] alg: No test for adler32 (adler32-zlib) [ 4064.586638] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4064.860829] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4066.552752] Key type lgssc registered [ 4067.868682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4076.054810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4076.088069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.554094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.058860] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4087.790382] LustreError: 141891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.807668] LustreError: 141891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4090.610831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.036918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4095.893811] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4104.192715] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4110.443161] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4117.784247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4118.157659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.287563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4124.401960] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4131.957413] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4135.604230] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4142.252545] Lustre: 144477:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4145.796264] Lustre: 144525:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4147.248638] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4148.185279] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4153.503701] Lustre: 144671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4154.242546] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4155.198336] Lustre: 144780:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4161.512267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4161.517375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.536567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4161.551039] Lustre: Skipped 1 previous similar message [ 4166.416692] Lustre: server umount lustre-OST0000 complete [ 4171.746396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.762376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4171.767395] Lustre: Skipped 2 previous similar messages [ 4176.232940] Lustre: server umount lustre-MDT0000 complete [ 4176.866278] LustreError: 141888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.887090] LustreError: 141888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4179.548212] LustreError: 141873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448338 with bad export cookie 10758208726716944862 [ 4179.557927] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4179.573283] LustreError: 141873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4180.009741] Lustre: server umount lustre-MDT0001 complete [ 4190.956279] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4194.468371] Key type lgssc unregistered [ 4194.976820] LNet: 145833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4194.997961] LNetError: 145833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4195.058846] LNet: Removed LNI 192.168.203.114@tcp [ 4196.200849] Key type .llcrypt unregistered [ 4196.209051] Key type ._llcrypt unregistered [ 4220.954280] Key type ._llcrypt registered [ 4220.958653] Key type .llcrypt registered [ 4221.109882] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4222.532538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4222.566129] alg: No test for adler32 (adler32-zlib) [ 4223.654619] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4223.949579] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4225.688222] Key type lgssc registered [ 4226.996702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4236.465053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4236.503243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.083438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.154317] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4247.527181] LustreError: 147248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.543221] LustreError: 147248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4252.596480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.656628] LustreError: 147249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.027212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4257.711064] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4266.579501] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.314741] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4279.016989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.469070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.547778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4285.800658] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4293.261179] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.925583] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4302.723341] Lustre: *** cfs_fail_loc=159, val=0*** [ 4302.728466] Lustre: Skipped 1 previous similar message [ 4318.895793] Lustre: lustre-MDT0000: Client 2e76bbcf-a2a1-4a58-a111-95fb88151146 (at 192.168.203.14@tcp) reconnecting [ 4339.169269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4339.176205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.178083] LustreError: Skipped 1 previous similar message [ 4339.183488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.218141] Lustre: Skipped 1 previous similar message [ 4343.141290] Lustre: server umount lustre-OST0000 complete [ 4349.412384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.415583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.443539] Lustre: Skipped 1 previous similar message [ 4349.465056] Lustre: Skipped 2 previous similar messages [ 4353.962858] Lustre: server umount lustre-MDT0000 complete [ 4354.533332] LustreError: 147249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.541224] LustreError: 147249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4357.933227] LustreError: 147228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448516 with bad export cookie 10265207363262704287 [ 4357.935255] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4357.954474] LustreError: 147228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4358.334253] Lustre: server umount lustre-MDT0001 complete [ 4369.396388] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4372.505509] Key type lgssc unregistered [ 4372.896849] LNet: 150923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4372.905660] LNetError: 150923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4372.922378] LNet: Removed LNI 192.168.203.114@tcp [ 4373.893241] Key type .llcrypt unregistered [ 4373.896311] Key type ._llcrypt unregistered [ 4398.292120] Key type ._llcrypt registered [ 4398.299653] Key type .llcrypt registered [ 4398.450152] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4399.855778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4399.881201] alg: No test for adler32 (adler32-zlib) [ 4400.979485] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4401.197786] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4402.850053] Key type lgssc registered [ 4404.160040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4412.510563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4412.525092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.013896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.811110] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4423.141086] LustreError: 152340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.155488] LustreError: 152340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4427.994938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.257119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4428.461734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4433.088196] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4440.200475] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4445.251964] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.530458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.848783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.950284] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4458.351407] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4459.062460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4464.978586] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4484.760227] Lustre: lustre-MDT0001: evicting 6afcb3de-08de-4fc5-b563-c03ce626d33d (at 192.168.203.14@tcp) by administrative request [ 4484.774367] Lustre: Skipped 1 previous similar message [ 4515.306791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4515.308070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.314097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.315555] LustreError: Skipped 1 previous similar message [ 4515.361988] Lustre: Skipped 1 previous similar message [ 4519.386057] Lustre: server umount lustre-OST0000 complete [ 4525.542968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.549972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4525.553662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.553671] Lustre: Skipped 1 previous similar message [ 4525.562105] Lustre: Skipped 1 previous similar message [ 4530.659339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.666439] Lustre: Skipped 2 previous similar messages [ 4535.779459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.784282] Lustre: Skipped 1 previous similar message [ 4537.313923] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4537.599365] Lustre: server umount lustre-MDT0000 complete [ 4540.904820] LustreError: 152334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.945545] LustreError: 152334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4541.676566] LustreError: 152319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448700 with bad export cookie 1660767717055479136 [ 4541.694149] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.200422] Lustre: server umount lustre-MDT0001 complete [ 4552.941342] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4556.102255] Key type lgssc unregistered [ 4556.430128] LNet: 156411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4556.446815] LNetError: 156411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4557.485695] LNet: Removed LNI 192.168.203.114@tcp [ 4558.480353] Key type .llcrypt unregistered [ 4558.484624] Key type ._llcrypt unregistered [ 4630.257869] Key type ._llcrypt registered [ 4630.259183] Key type .llcrypt registered [ 4630.343875] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4631.318679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4631.340504] alg: No test for adler32 (adler32-zlib) [ 4632.379510] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4632.553221] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4634.184233] Key type lgssc registered [ 4635.248473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4642.004952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4642.028392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.488292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.266262] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4654.016129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.051598] LustreError: 158880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.066859] LustreError: 158880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4654.446452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4658.279351] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4665.656368] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4670.401063] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4676.007575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.302898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4679.413995] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4681.716783] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4688.324706] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4689.670216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4700.133975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4700.144778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.154947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4700.161651] Lustre: Skipped 1 previous similar message [ 4703.106731] Lustre: server umount lustre-OST0000 complete [ 4710.373736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.387521] Lustre: Skipped 1 previous similar message [ 4710.391029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4710.395506] Lustre: Skipped 1 previous similar message [ 4715.490648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.611253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.615476] Lustre: Skipped 2 previous similar messages [ 4721.120125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4721.290724] Lustre: server umount lustre-MDT0000 complete [ 4724.882179] LustreError: 158863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448883 with bad export cookie 5849257227566778187 [ 4724.891714] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4724.902373] LustreError: 158863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4725.181682] Lustre: server umount lustre-MDT0001 complete [ 4734.998744] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4737.520620] Key type lgssc unregistered [ 4737.753065] LNet: 162425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.759492] LNetError: 162425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.775564] LNet: Removed LNI 192.168.203.114@tcp [ 4738.566468] Key type .llcrypt unregistered [ 4738.568876] Key type ._llcrypt unregistered [ 4760.784621] Key type ._llcrypt registered [ 4760.786953] Key type .llcrypt registered [ 4760.881520] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4761.877215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4761.973226] alg: No test for adler32 (adler32-zlib) [ 4763.019785] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4763.262116] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4764.936254] Key type lgssc registered [ 4766.153093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4779.827371] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:22:16 (1788448936) [ 4794.305118] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 4803.455693] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4840.979886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4841.003561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.168937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4842.189583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4842.245832] Lustre: lustre-MDT0000: new disk, initializing [ 4842.329427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.352380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4846.792657] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4856.295675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.396647] Lustre: 166711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4856.462873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4856.468214] Lustre: Skipped 1 previous similar message [ 4856.542586] Lustre: lustre-MDT0001: new disk, initializing [ 4856.648283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4856.694540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4856.708601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4861.277959] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4870.602383] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4875.881811] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4882.736441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4882.989913] Lustre: lustre-OST0000: new disk, initializing [ 4882.995910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4883.003815] Lustre: 168682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4883.104763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.199573] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4892.760446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4892.770342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4892.908257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4899.264544] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4913.633501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4913.639418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.657477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.678449] Lustre: Skipped 1 previous similar message [ 4915.642974] Lustre: server umount lustre-OST0000 complete [ 4923.364310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4923.376659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.397414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4923.409753] Lustre: Skipped 1 previous similar message [ 4928.993123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4929.002940] Lustre: Skipped 2 previous similar messages [ 4933.091920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.101437] Lustre: Skipped 1 previous similar message [ 4934.112786] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4934.453475] Lustre: server umount lustre-MDT0000 complete [ 4937.906242] LustreError: 167518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449096 with bad export cookie 18357960494408614808 [ 4937.909508] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4937.922193] LustreError: 167518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4938.198965] Lustre: server umount lustre-MDT0001 complete [ 4950.369084] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 4953.444265] Key type lgssc unregistered [ 4953.779904] LNet: 170591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4953.799169] LNetError: 170591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4953.832206] LNet: Removed LNI 192.168.203.114@tcp [ 4954.564970] Key type .llcrypt unregistered [ 4954.570126] Key type ._llcrypt unregistered [ 4978.651551] Key type ._llcrypt registered [ 4978.656458] Key type .llcrypt registered [ 4978.778089] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 4980.003992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4980.034293] alg: No test for adler32 (adler32-zlib) [ 4981.043203] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4981.254300] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 4982.912239] Key type lgssc registered [ 4983.992485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4991.720789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4991.735742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.195405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.306458] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5003.756608] LustreError: 172005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.789573] LustreError: 172005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5005.083900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.454663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5010.515890] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5019.022805] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5025.101483] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5031.184892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.335120] LustreError: 173825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.491684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5037.062753] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 5038.579383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5044.505939] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5050.009985] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5050.034673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.050543] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5050.061687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.077619] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5051.077905] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5051.105874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.127949] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5051.139028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.159646] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5052.162548] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5052.197203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.210084] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5052.220108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.229891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5053.101901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.076450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.843288] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5054.848837] Lustre: Skipped 2 previous similar messages [ 5054.868022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.881981] Lustre: Skipped 2 previous similar messages [ 5054.887603] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5054.899311] Lustre: Skipped 2 previous similar messages [ 5054.902903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.918231] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5054.928081] Lustre: Skipped 2 previous similar messages [ 5055.663876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.506848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.596118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.709950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.905970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5059.914613] Lustre: Skipped 4 previous similar messages [ 5059.952609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.966139] Lustre: Skipped 4 previous similar messages [ 5059.974779] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5059.979779] Lustre: Skipped 4 previous similar messages [ 5059.983530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.993112] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5060.002202] Lustre: Skipped 4 previous similar messages [ 5060.987813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.930130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.804272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.682685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.611442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.548144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.402098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.212438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.076992] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5068.081180] Lustre: Skipped 8 previous similar messages [ 5068.108479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.119745] Lustre: Skipped 8 previous similar messages [ 5068.125101] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5068.130792] Lustre: Skipped 8 previous similar messages [ 5068.135531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.143759] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5068.152037] Lustre: Skipped 8 previous similar messages [ 5068.976198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.060798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.889410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.849769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.725567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.459469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.197521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.975602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.989435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.847389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.795884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.791893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.899598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.877970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.037233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.179233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.958898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.867684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5084.875830] Lustre: Skipped 17 previous similar messages [ 5084.880968] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5084.893729] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5084.899571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.923122] Lustre: Skipped 17 previous similar messages [ 5084.941779] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5084.946598] Lustre: Skipped 17 previous similar messages [ 5084.954408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.961295] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5084.966520] Lustre: Skipped 17 previous similar messages [ 5085.693583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.397939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.254348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.225712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.077708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.176971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.196906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.208481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.903576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.824725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.668043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.447323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.147025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.860901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.568739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.456631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.299880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.266244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.204762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.086177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.023971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.796507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.539984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.525222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.402144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.338195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.211143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.139407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.006233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.817214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.663199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.695072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.680993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.697996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.805515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.856578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.725401] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5117.731491] Lustre: Skipped 36 previous similar messages [ 5117.760644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.768691] Lustre: Skipped 36 previous similar messages [ 5117.775852] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5117.781420] Lustre: Skipped 36 previous similar messages [ 5117.785261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.818514] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5117.821575] Lustre: Skipped 36 previous similar messages [ 5118.688857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.724567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.516621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.425809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.284307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.112883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.986631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.882224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.853896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.612287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.516077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.521895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.476620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.498791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.443535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.379000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.226229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.087160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.849424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.755711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.561683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.308903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.026188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.803077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.551788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.425571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.128392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.978484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.669530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.397519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.194351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.959857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.760453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.527654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.203094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.997405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.703597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.584361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.405274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.343055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.199836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.956800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.667566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.519689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.482964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.587190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.437775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.213982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.063351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.897449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.804083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.675892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.533063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.358555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.160526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.131577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.016189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.872324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.696057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.546493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.465823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.416515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.159995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.979821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.727204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.502703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.244153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.182396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.947679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.742224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.540951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.392714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.063137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.919370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.694991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.484100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.264803] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5182.269330] Lustre: Skipped 76 previous similar messages [ 5182.290181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.304278] Lustre: Skipped 76 previous similar messages [ 5182.312572] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5182.326226] Lustre: Skipped 76 previous similar messages [ 5182.334938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.356317] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5182.362627] Lustre: Skipped 76 previous similar messages [ 5183.266376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.107142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.884294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.589660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.438316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.245550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.974240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.675410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.473831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.182785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.984549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.724104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.556503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.347849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.190514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.941112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.766036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.570278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.391513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.111781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.744406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.534607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.328094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.105641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.921046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.708137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.613818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.392325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.245734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.139160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.891503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.719453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.627657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.401477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.348157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.151820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.982256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.689985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.535578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.266495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.144766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.923059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.712954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.647694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.622464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.461546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.404392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.247489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.174886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.987906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.834048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.828712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.741455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.700195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.613485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.596273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.545501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.770100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.703655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.816641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.712135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.599862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.458659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.434332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.347316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.114562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.037341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.047174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.217357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.938964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.929080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.885487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.920951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.781447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.603251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.699055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.663518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.455838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.455396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.852894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.971330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.107188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.528991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.731719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.734191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.907051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.854495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.836854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.737686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.825444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.825348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.947194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.933342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.859910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.778104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.753517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.637684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.772236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.774843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.747729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.655038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.587955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.438563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.533691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.554323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.509654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.445415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.456932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.233818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.118645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.022472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.871719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.692778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.562667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.376446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.290069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.274024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.058713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.773993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.735188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.473715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.239592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.904754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.655909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.489461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.574789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.479557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.247331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.398367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.427339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.379699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.297932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.430428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.514925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.452730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.465883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.579779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.514329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.696116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.654918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.578832] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5310.595646] Lustre: Skipped 140 previous similar messages [ 5310.623724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.645027] Lustre: Skipped 140 previous similar messages [ 5310.650445] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5310.663554] Lustre: Skipped 140 previous similar messages [ 5310.665572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.697041] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5310.708165] Lustre: Skipped 140 previous similar messages [ 5311.662765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.482265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.562326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.615809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.641310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.511903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.579486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.759987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.753808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.789642] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5320.802520] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5320.809684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.581262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.360479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.342808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.383139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.374471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.545271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.541088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.478487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.428685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.493329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.477211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.380313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.186790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.077430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.909282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.901201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.916728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.877252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.673274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.943111] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5339.956734] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5339.974394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.877449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.891104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.731966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.731688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.560722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.408148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.196100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.170270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.957288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.767405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.610169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.449444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.330327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.153356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.027524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.031453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.942399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.745777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.459707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.366181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.232593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.081730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.953304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.754954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.630861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.459778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.263863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.109703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.969100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.716283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.517640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.333654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.111340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.901976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.643136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.499648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.429549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.484485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.500393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.568928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.471607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.554301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.427819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.223216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.194164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.038987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.905574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.858354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.681957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.721085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.677683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.616276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.461105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.431936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.222751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.093367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.897169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.738722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.650240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.543320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.356236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.223593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.012658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.914116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.849882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.650985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.397542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.365400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.119476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.943299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.858821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.775713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.615685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.496536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.385726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.306165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.380048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.358714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.312991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.436614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.172826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.036500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.836682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.723303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.619938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.611483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.394285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.275192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.100095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.998688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.778891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.579443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.487461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.373087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.172162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.022150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.812912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.714112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.647424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.745787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.653845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.444196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.157585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.081395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.798442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.554984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.440551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.476829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.338834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.316252] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5436.331857] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5436.332180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.206485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.062405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.940820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.901404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.636522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.544432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.581845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.755342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.890661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.735376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.632867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.657418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.887581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.098574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.216509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.105656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.021910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.856447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.820907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.622749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.546224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.393091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.453167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.323549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.079599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.828820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.623976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.431114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.393453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.271093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.113039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.069859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.000104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.808805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.724724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.487708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.466228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.634317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.687579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.599528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.608895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.666989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.551600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.778890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.058227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.369363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.617025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.702633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.599186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.622432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.447213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.373318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.377648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.261307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.304218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.267428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.055420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.307507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.127853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.138389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.281981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.337328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.231598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.060507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.143748] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5499.164151] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5499.183869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.187715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.079390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.178090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.157758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.396863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.663255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.476210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.595173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.822637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.786586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.689813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.660808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.781325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.814679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.604886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.281566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.995818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.712022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.662394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.799086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.110699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.048479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.264418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.574878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.744793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.068259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.393441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.395656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.315343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.411768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.742966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.956370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.092278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.159913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.225071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.182711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.136240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.209509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.191856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.324946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.487718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.501521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.647310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.838149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.574369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.384646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.516929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.537881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.318431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.202954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.047971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.059287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.040310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.019294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.985959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.034499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.908374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.878363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.685628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.564068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.184468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.976645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.760437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.830780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.704283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.504788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.432556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.270877] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5567.282228] Lustre: Skipped 272 previous similar messages [ 5567.305257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.318348] Lustre: Skipped 272 previous similar messages [ 5567.322955] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5567.328543] Lustre: Skipped 272 previous similar messages [ 5567.331824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.342785] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5567.357693] Lustre: Skipped 272 previous similar messages [ 5568.298037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.251561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.100538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.119549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.027512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.033962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.902242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.661601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.538100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.577896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.473469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.535627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.313676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.318080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.313733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.676938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.812800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.881908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.966829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.043957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.218806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.269296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.154625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.382981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.418665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.745862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.206803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.300074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.244025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.379667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.334543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.434749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.199326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.475901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.728621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.653834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.840177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.974488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.156174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.473391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.544911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.651916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.592251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.567235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.308493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.073443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.235074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.225815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.333767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.432562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.458319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.464885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.495662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.710745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.822481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.759216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.726137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.638421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.686217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.722215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.673109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.654391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.527396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.519488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.443864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.378314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.273657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.140626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.891852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.888990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.750376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.495466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.209564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.011297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.855977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.858366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.673170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.525566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.318049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.208128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.999243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.888291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.644662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.442403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.353681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.330096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.122244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.067536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.897626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.685021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.545826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.353465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.218026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.095101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.101759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.258329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.193700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.062892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.013307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.064861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.176096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.196248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.221990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.189367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.374945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.476567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.694299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.829811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.894862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.935820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.083168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.374987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.630022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.585032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.712097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.970285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.905630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.995366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.783949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.991475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.844048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.660386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.485921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.451655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.287960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.207101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.239499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.180400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.105756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.079327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.018533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.921533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.711771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.668158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.633444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.588447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.466138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.349610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.096864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.922449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.670778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.564273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.311296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.119398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.920884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.826426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.763259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.792540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.735338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.726798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.709431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.515825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.348630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.230466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.037791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.224311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.975882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.854256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.734072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.669422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.427654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.406934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.106648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.973301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.972708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.805838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.647306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.474976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.263619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.175488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.121305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.014321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.858826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.822408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.667371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.636947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.392820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.145861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.196233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.167313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.041584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.122653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.116753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.269225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.285899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.509960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.458687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.429785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.597295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.611493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.594592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.456680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.360726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.297055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.258535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.179503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.115293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.214529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.185335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.215472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.017501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.286515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.291200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.194557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.098871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.029863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.841350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.806768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.754519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.723262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.789264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.863847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.067697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.008209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.922400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.772127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.670325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.657096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.626388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.674860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.753161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.510996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.437457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.336455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.233161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.094883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.009628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.035194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.959288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.930165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.924580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.825319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.720327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.713324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.608057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.550404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.402850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.250110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.171644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.020629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.860561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.867940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.730618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.495200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.305711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.210456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.002074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.875050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.730159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.530181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.384031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.270222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.956948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.660481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.540247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.439912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.242337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.059645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.894765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.932434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.789200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.680104] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5816.687805] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5816.702239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.653958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.538082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.522772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.463316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.383186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.407733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.439732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.657207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.911267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.968180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.885830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.083042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.138927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.070534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.296431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.766967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.083270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.361128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.651775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.015963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.961952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.066642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.475961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.665345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.871597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.987694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.219814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.191496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.405223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.468155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.451967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.403475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.432559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.430306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.385971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.402601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.303960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.126167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.158778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.092754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.114707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.991326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.969850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.842186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.058665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.303591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.263628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.220173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.045280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.918587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.877542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.801960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.700789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.602714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.520258] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5874.528875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.535946] LustreError: 172025:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5875.446440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.386825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.254727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.027825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.841091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.649140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.572486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.428740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.349689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.292169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.156280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.030313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.863209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.754811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.630789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.411248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.292331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.386216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.225377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.183320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.010994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.960983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.811037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.703111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.507328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.365504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.223374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.103202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.112539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.037122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.918704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.659079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.551926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.571383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.475640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.373198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.324558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.398398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.216972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.052937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.064631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.152172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.150763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.163237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.411900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.345170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.200947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.070853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.931800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.996057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.253273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.420711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.579286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.746338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.790331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.821743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.967519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.933352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.264988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.247493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.471613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.486149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.421488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.517673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.408639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.570711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.755361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.794437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.904844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.816761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.740922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.649207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.718917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.682623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.735890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.853374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.815498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.957719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.000230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.289087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.478243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.441384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.569311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.554512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.527888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.526327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.524418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.435670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.498912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.537352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.575879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.610926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.673543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.699713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.703634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.680141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.969637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.933214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.015914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.150350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.129484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.298062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.074278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.033193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.865531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.799838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.765511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.069436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.189424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.516105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.299080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.406271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.400346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.426038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.571586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.499707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.668384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.318428] LustreError: 173827:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6345.329355] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6345.346519] LustreError: 172025:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 6818.279081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.288485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.303559] Lustre: Skipped 434 previous similar messages [ 6818.306916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.322104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6820.335457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.348993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.355142] Lustre: Skipped 1 previous similar message [ 6823.900922] Lustre: server umount lustre-OST0000 complete [ 6828.513125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6828.523385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6836.706583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6836.715029] Lustre: Skipped 3 previous similar messages [ 6841.317155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6841.835374] LustreError: 172001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6841.849599] LustreError: 172001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6841.855286] Lustre: server umount lustre-MDT0000 complete [ 6845.893548] LustreError: 171985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451004 with bad export cookie 10076091236459853505 [ 6845.895422] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.907059] LustreError: 171985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6846.288833] Lustre: server umount lustre-MDT0001 complete [ 6857.910627] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6861.822978] Key type lgssc unregistered [ 6862.232697] LNet: 227227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6862.241692] LNetError: 227227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6862.272805] LNet: Removed LNI 192.168.203.114@tcp [ 6863.426857] Key type .llcrypt unregistered [ 6863.431235] Key type ._llcrypt unregistered [ 6887.756470] Key type ._llcrypt registered [ 6887.759786] Key type .llcrypt registered [ 6887.921243] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6888.674731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6888.739300] alg: No test for adler32 (adler32-zlib) [ 6889.915797] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6890.313367] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6892.072637] Key type lgssc registered [ 6893.304720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6900.922975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6900.934095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.402994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.978386] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6913.000456] LustreError: 228645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6913.012962] LustreError: 228645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6915.387569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.797118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6920.144825] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6927.168489] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6932.036738] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6937.681523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.944837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.248777] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6947.190125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6949.380238] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.569657] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6956.124880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.136440] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6956.147990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6956.167188] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6956.167265] LustreError: 228664:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6961.638222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6961.652456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.673631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6962.354146] Lustre: lustre-OST0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 6962.662852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.434161] Lustre: lustre-OST0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 6967.447989] Lustre: Skipped 2 previous similar messages [ 6972.384164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6972.504455] Lustre: server umount lustre-OST0000 complete [ 6976.200169] LustreError: 228664:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7005.976575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.383726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.504782] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7017.642723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7017.669736] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7019.741236] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.048593] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7022.836483] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7022.853649] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7027.039272] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b91c4f66800.ost_server_uuid 50 [ 7028.775130] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b91c4f66800.ost_server_uuid in FULL state after 0 sec [ 7033.314423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7033.315339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.319301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7033.319308] Lustre: Skipped 2 previous similar messages [ 7033.355038] Lustre: Skipped 1 previous similar message [ 7038.446330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7038.454923] Lustre: Skipped 2 previous similar messages [ 7039.006606] Lustre: server umount lustre-OST0000 complete [ 7043.556821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.472417] Lustre: server umount lustre-MDT0000 complete [ 7048.674974] LustreError: 228644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.689038] LustreError: 228644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7052.441744] LustreError: 228625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451211 with bad export cookie 16983541046799738310 [ 7052.445032] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7052.468633] LustreError: 228625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7052.957164] Lustre: server umount lustre-MDT0001 complete [ 7063.344298] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7066.027807] Key type lgssc unregistered [ 7066.362838] LNet: 233858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7066.375745] LNetError: 233858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7067.435312] LNet: Removed LNI 192.168.203.114@tcp [ 7068.184170] Key type .llcrypt unregistered [ 7068.187053] Key type ._llcrypt unregistered [ 7096.629538] Key type ._llcrypt registered [ 7096.634185] Key type .llcrypt registered [ 7096.739822] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7097.541734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7097.578092] alg: No test for adler32 (adler32-zlib) [ 7098.667304] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7098.992883] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7100.656172] Key type lgssc registered [ 7101.773603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7109.965996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7109.983082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7111.429666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.227619] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7121.903491] LustreError: 235598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7121.925226] LustreError: 235598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7122.708502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.015923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7126.874534] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7130.154538] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7135.780887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.085855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7137.136064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7141.289358] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7148.313658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.485798] Lustre: lustre-OST0001: new disk, initializing [ 7148.489789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7148.495448] Lustre: 237904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7148.580481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7154.282343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7154.295471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7154.409571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7154.558782] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7165.915941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7173.693456] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7189.473293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7189.492272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7189.511077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7190.497666] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.502819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7190.528876] Lustre: Skipped 2 previous similar messages [ 7190.534916] Lustre: Skipped 2 previous similar messages [ 7195.631105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7195.636759] Lustre: Skipped 3 previous similar messages [ 7200.224438] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7200.449896] Lustre: server umount lustre-MDT0000 complete [ 7200.746429] LustreError: 235594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.772741] LustreError: 235594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7203.743743] LustreError: 237903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451362 with bad export cookie 708785021689397617 [ 7203.753936] LustreError: 237903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7203.756627] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.105409] Lustre: server umount lustre-MDT0001 complete [ 7217.343443] Lustre: server umount lustre-OST0000 complete [ 7231.263646] Lustre: server umount lustre-OST0001 complete [ 7252.468764] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7263.275214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.724862] LustreError: 241015:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.801775] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.579262] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7268.846897] LustreError: 241016:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.961629] LustreError: 241015:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7275.697269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.197111] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.311915] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7284.516480] Lustre: 242158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7292.073591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7292.375759] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.967252] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7301.618110] LustreError: 242515:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.643961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7308.771472] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 7315.760934] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7324.215120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7327.662571] Lustre: 244029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7337.447469] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7337.460819] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.479157] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7339.498899] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.518604] Lustre: Skipped 1 previous similar message [ 7343.507429] Lustre: server umount mylustre-MDT0000 complete [ 7344.612786] LustreError: 241015:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.628238] LustreError: 241015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7347.261628] LustreError: 240991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451506 with bad export cookie 708785021689400431 [ 7347.269964] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.277953] LustreError: 240991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7347.695783] Lustre: server umount mylustre-MDT0001 complete [ 7362.006988] Lustre: server umount mylustre-OST0000 complete [ 7365.100069] Lustre: 234851:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451508/real 1788451508] req@ffff924d3c5cd180 x1875327065413120/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7365.127308] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.141167] Lustre: Skipped 1 previous similar message [ 7365.797350] Lustre: server umount mylustre-OST0001 complete [ 7386.373405] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7395.839860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7396.314843] LustreError: 246337:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.374954] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7396.380106] Lustre: Skipped 1 previous similar message [ 7400.581458] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7408.846950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7413.561692] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7416.530312] Lustre: 247478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7422.517674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.243570] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7437.535707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7441.855178] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49792) [ 7441.855337] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7444.192673] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7451.869354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7455.183528] Lustre: 249352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7462.369432] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.374236] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7462.391046] Lustre: Skipped 3 previous similar messages [ 7462.397092] Lustre: Skipped 6 previous similar messages [ 7464.365473] Lustre: server umount tfs-MDT0000 complete [ 7467.490980] LustreError: 246333:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7467.508625] LustreError: 246333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7467.870381] LustreError: 246313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451626 with bad export cookie 708785021689402097 [ 7467.874811] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7467.885929] LustreError: 246313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7468.335320] Lustre: server umount tfs-MDT0001 complete [ 7482.985904] Lustre: server umount tfs-OST0000 complete [ 7497.496541] Lustre: server umount tfs-OST0001 complete [ 7516.854221] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7527.033838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.609394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.622480] Lustre: Skipped 3 previous similar messages [ 7531.546922] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7540.727978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.739435] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7550.424526] Lustre: 252750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7557.379706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.998425] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7572.161347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49824) [ 7572.290578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.596936] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7579.184857] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7587.047590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7590.658601] Lustre: 254619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7598.293144] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:09:15 (1788451755) [ 7618.018911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7618.023341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.034722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.043531] Lustre: Skipped 1 previous similar message [ 7628.256154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7628.467904] Lustre: server umount lustre-MDT0000 complete [ 7628.775149] LustreError: 251608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.793603] LustreError: 251608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7632.268081] LustreError: 252353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451791 with bad export cookie 708785021689403763 [ 7632.272749] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7632.286873] LustreError: 252353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7637.932817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7637.946384] Lustre: Skipped 10 previous similar messages [ 7638.897455] Lustre: server umount lustre-MDT0001 complete [ 7648.689736] Lustre: server umount lustre-OST0000 complete [ 7658.393432] Lustre: server umount lustre-OST0001 complete [ 7664.923934] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7675.009504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7710.718634] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 7710.718634] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 7710.718634] [ 7710.736254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 7710.978953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7711.024273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7711.148250] Lustre: lustre-MDT0000: new disk, initializing [ 7711.269468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7711.278573] Lustre: Skipped 3 previous similar messages [ 7711.296415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7715.458326] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7726.037580] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 7726.037580] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 7726.037580] [ 7726.059519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 7726.189801] Lustre: 257805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7726.243852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7726.248200] Lustre: Skipped 1 previous similar message [ 7726.327571] Lustre: lustre-MDT0001: new disk, initializing [ 7726.449505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7726.456583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7730.764404] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7740.808620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7740.991507] Lustre: lustre-OST0000: new disk, initializing [ 7740.994495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7740.998629] Lustre: 259390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7742.529216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7742.556204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7742.681922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7747.197377] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7757.539607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.731143] Lustre: lustre-OST0001: new disk, initializing [ 7757.738882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7757.745532] Lustre: 260260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7758.887908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7758.903397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7759.040248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7765.354773] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7780.090207] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:12:16 (1788451936) [ 7799.483886] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:12:36 (1788451956) [ 7814.630405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7814.630683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7814.645907] LustreError: Skipped 1 previous similar message [ 7814.648807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7814.661048] Lustre: Skipped 6 previous similar messages [ 7819.926887] Lustre: server umount lustre-OST0000 complete [ 7823.855635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7829.706233] Lustre: server umount lustre-MDT0000 complete [ 7833.308856] LustreError: 257797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451992 with bad export cookie 708785021689405317 [ 7833.316728] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7833.328339] LustreError: 257797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7839.877303] Lustre: server umount lustre-MDT0001 complete [ 7851.388857] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7924.799974] Lustre: server umount lustre-OST0001 complete [ 7931.260934] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7945.676389] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7978.598665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.841261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7978.904327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7979.041151] Lustre: lustre-MDT0000: new disk, initializing [ 7979.142185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7979.154165] Lustre: Skipped 3 previous similar messages [ 7979.170044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7983.790540] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7994.045648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.171340] Lustre: 265060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7994.568234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7999.041444] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8008.985108] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8015.968883] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8022.967350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8023.272814] Lustre: lustre-OST0000: new disk, initializing [ 8023.279041] Lustre: Skipped 1 previous similar message [ 8023.289734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8023.296423] Lustre: Skipped 2 previous similar messages [ 8023.300698] Lustre: 267028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8024.589654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8024.605313] Lustre: Skipped 1 previous similar message [ 8024.614589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8024.790752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8029.947412] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8040.725232] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8050.665060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8050.666187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.668709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8050.668721] Lustre: Skipped 9 previous similar messages [ 8050.671727] LustreError: Skipped 1 previous similar message [ 8050.703516] Lustre: Skipped 5 previous similar messages [ 8054.570939] Lustre: server umount lustre-OST0000 complete [ 8064.794151] Lustre: server umount lustre-MDT0000 complete [ 8066.020948] LustreError: 265944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.036359] LustreError: 265944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8068.118614] LustreError: 265054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452227 with bad export cookie 708785021689406787 [ 8068.123509] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8068.142232] LustreError: 265054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8068.542768] Lustre: server umount lustre-MDT0001 complete [ 8078.856145] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8081.666827] Key type lgssc unregistered [ 8081.934735] LNet: 268794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8081.946925] LNetError: 268794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8082.993576] LNet: Removed LNI 192.168.203.114@tcp [ 8083.835159] Key type .llcrypt unregistered [ 8083.837183] Key type ._llcrypt unregistered [ 8108.115673] Key type ._llcrypt registered [ 8108.120526] Key type .llcrypt registered [ 8108.223371] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8109.071202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8109.230218] alg: No test for adler32 (adler32-zlib) [ 8110.287147] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8110.511088] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8112.162345] Key type lgssc registered [ 8113.293262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8120.547747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8120.571953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.091116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.654320] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8132.580188] LustreError: 270254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.623467] LustreError: 270254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8135.583559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8136.112318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8142.277360] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8150.458405] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8157.232789] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8165.841681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.460201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.479332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8173.101686] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8181.839840] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8182.812495] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8185.833184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.839823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8185.864584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8190.983192] Lustre: server umount lustre-OST0000 complete [ 8196.075534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8196.094722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8196.097106] Lustre: Skipped 2 previous similar messages [ 8196.117771] Lustre: Skipped 2 previous similar messages [ 8201.187315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8201.196951] Lustre: Skipped 1 previous similar message [ 8201.515452] Lustre: server umount lustre-MDT0000 complete [ 8205.281630] LustreError: 270254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.305454] LustreError: 270254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8205.725424] LustreError: 272099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452364 with bad export cookie 11285863712916750172 [ 8205.737740] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8205.740735] LustreError: 272099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8206.295689] Lustre: server umount lustre-MDT0001 complete [ 8217.143305] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8219.686689] Key type lgssc unregistered [ 8219.951955] LNet: 273842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8219.965850] LNetError: 273842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8220.000844] LNet: Removed LNI 192.168.203.114@tcp [ 8220.904275] Key type .llcrypt unregistered [ 8220.912093] Key type ._llcrypt unregistered [ 8236.868635] Key type ._llcrypt registered [ 8236.871470] Key type .llcrypt registered [ 8236.964383] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8237.834970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8237.869071] alg: No test for adler32 (adler32-zlib) [ 8239.013094] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8239.344202] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8241.000288] Key type lgssc registered [ 8242.145171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8250.364154] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8250.383315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8265.952518] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8271.072572] LustreError: 275036:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 8297.760289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8304.015722] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8311.401925] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8313.767453] Lustre: server umount lustre-OST0000 complete [ 8331.851219] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8334.613922] Key type lgssc unregistered [ 8334.919281] LNet: 276670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8334.931293] LNetError: 276670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8335.981960] LNet: Removed LNI 192.168.203.114@tcp [ 8337.051487] Key type .llcrypt unregistered [ 8337.054445] Key type ._llcrypt unregistered [ 8374.479348] Key type ._llcrypt registered [ 8374.483940] Key type .llcrypt registered [ 8374.607236] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8375.870929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8375.951817] alg: No test for adler32 (adler32-zlib) [ 8377.005975] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8377.365717] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8379.082332] Key type lgssc registered [ 8380.394107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8393.289334] loop0: detected capacity change from 0 to 204800000 [ 8393.462403] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8393.486543] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8393.828849] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8394.983595] loop0: detected capacity change from 0 to 204800000 [ 8395.225250] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8395.236794] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8395.475908] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8396.603743] loop0: detected capacity change from 0 to 204800000 [ 8396.837748] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8396.849567] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8397.103817] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8398.425182] loop0: detected capacity change from 0 to 204800000 [ 8398.771347] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8398.793265] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8399.008389] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8400.271642] loop0: detected capacity change from 0 to 204800000 [ 8400.291204] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8405.139526] loop0: detected capacity change from 0 to 204800000 [ 8405.180778] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8409.658671] loop0: detected capacity change from 0 to 204800000 [ 8409.692454] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8413.653132] loop0: detected capacity change from 0 to 204800000 [ 8413.686611] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8417.815677] loop0: detected capacity change from 0 to 204800000 [ 8417.898317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8417.982994] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.039203] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8418.047385] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8418.056687] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8418.065435] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8418.075150] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8418.085405] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8418.096122] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8418.106253] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8418.115686] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8418.129247] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8418.140176] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8418.153340] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8418.167919] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8418.177017] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8424.154820] Lustre: server umount MGS complete [ 8425.157621] loop0: detected capacity change from 0 to 204800000 [ 8425.354347] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.288662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.348683] Lustre: lustre-MDT0000: Aborting client recovery [ 8427.201137] loop1: detected capacity change from 0 to 204800000 [ 8427.313094] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.446079] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8427.466316] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8427.477698] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8427.488372] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8427.497525] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8427.507409] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8427.515103] Lustre: 280518:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8427.542901] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8427.802762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.886953] Lustre: lustre-MDT0001: Aborting client recovery [ 8428.738747] loop2: detected capacity change from 0 to 204800000 [ 8428.882256] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8428.963567] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8428.974937] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8428.983932] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8428.998625] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8429.014056] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8429.032893] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8429.043923] Lustre: 280596:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8429.168348] Lustre: Found index 0 for lustre-OST0000, updating log [ 8429.195283] Lustre: 280615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8429.263798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.287419] Lustre: lustre-OST0000: Aborting recovery [ 8430.160372] loop3: detected capacity change from 0 to 204800000 [ 8430.348589] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.430900] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8430.446514] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8430.459622] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8430.467250] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8430.480988] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8430.490948] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8430.499920] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8430.528806] Lustre: Found index 1 for lustre-OST0001, updating log [ 8430.570216] Lustre: 280687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8430.666059] Lustre: lustre-OST0001: Aborting recovery [ 8435.751182] LustreError: 280462:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8436.732564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8436.739918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8436.827512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8436.830776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8437.737383] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8437.745057] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8438.710318] LustreError: 280428:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 280us: rc = -2 [ 8439.258488] LustreError: 280428:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 548us: rc = -2 [ 8439.281267] LustreError: 280428:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 8453.089262] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8453.091251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8453.107167] Lustre: Skipped 3 previous similar messages [ 8453.113403] Lustre: Skipped 3 previous similar messages [ 8458.210545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.218177] Lustre: Skipped 3 previous similar messages [ 8458.958227] Lustre: server umount lustre-MDT0000 complete [ 8459.945056] LustreError: 280412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452618 with bad export cookie 15510479425277718543 [ 8459.946313] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.961020] LustreError: 280412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8463.329239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.330833] LustreError: 280623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.348822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8463.366422] LustreError: 280623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8463.387870] Lustre: Skipped 1 previous similar message [ 8464.353267] LustreError: 280427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.370293] LustreError: 280427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8466.476763] Lustre: server umount lustre-MDT0001 complete [ 8473.639983] Lustre: server umount lustre-OST0000 complete [ 8480.747509] Lustre: server umount lustre-OST0001 complete [ 8488.651453] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 12:24:05 (1788452645) [ 8515.413186] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8524.311118] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8556.807673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.078696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8557.126322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8557.211386] Lustre: lustre-MDT0000: new disk, initializing [ 8557.309758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.313694] Lustre: Skipped 1 previous similar message [ 8557.337095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8561.450769] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8571.683141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.823371] Lustre: 284752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8571.875823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8571.881220] Lustre: Skipped 1 previous similar message [ 8571.956632] Lustre: lustre-MDT0001: new disk, initializing [ 8572.065594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.100790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8572.122265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8576.312475] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8585.445991] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8590.764394] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8597.280507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8597.578714] Lustre: lustre-OST0000: new disk, initializing [ 8597.582745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8597.587977] Lustre: 286720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8597.693918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.225054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8604.238911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8604.396075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8604.797143] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8615.116278] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8620.745234] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8633.885147] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8641.844241] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8691.682336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.692877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.708695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.720089] Lustre: Skipped 2 previous similar messages [ 8691.729963] Lustre: Skipped 3 previous similar messages [ 8699.875785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8699.889769] Lustre: Skipped 3 previous similar messages [ 8704.993106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8705.199394] Lustre: server umount lustre-OST0000 complete [ 8710.628184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8710.647224] LustreError: Skipped 1 previous similar message [ 8710.661224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8710.682861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8710.691393] Lustre: Skipped 2 previous similar messages [ 8715.808244] Lustre: server umount lustre-MDT0000 complete [ 8718.310430] LustreError: 284762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.338587] LustreError: 284762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8719.578030] LustreError: 284743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452878 with bad export cookie 15510479425277721245 [ 8719.581577] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8719.596022] LustreError: 284743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8719.865314] Lustre: server umount lustre-MDT0001 complete [ 8729.202707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.477423] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8738.199266] Lustre: server umount MGS complete [ 8750.405312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.058473] LustreError: 290638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.107158] LustreError: 290651:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8751.219634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.208887] LustreError: 290639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8758.105154] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8766.436074] LustreError: 290638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.470099] LustreError: 290638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8768.439586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.423481] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8781.290852] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8788.409418] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8797.349625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.751584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.757079] Lustre: Skipped 1 previous similar message [ 8805.967064] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8813.950762] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8857.061032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8857.078226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8857.103734] Lustre: Skipped 1 previous similar message [ 8857.115563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8857.122760] Lustre: Skipped 2 previous similar messages [ 8862.645996] Lustre: server umount lustre-OST0000 complete [ 8867.298970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8867.332283] Lustre: Skipped 2 previous similar messages [ 8873.366898] Lustre: server umount lustre-MDT0000 complete [ 8877.541106] LustreError: 291367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8877.567590] LustreError: 291367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8878.268161] LustreError: 290619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453037 with bad export cookie 15510479425277723072 [ 8878.273317] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8878.276088] LustreError: 290619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8878.882214] Lustre: server umount lustre-MDT0001 complete [ 8892.002566] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8895.147535] Key type lgssc unregistered [ 8895.534828] LNet: 294612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8895.543748] LNetError: 294612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8895.563338] LNet: Removed LNI 192.168.203.114@tcp [ 8896.473038] Key type .llcrypt unregistered [ 8896.477613] Key type ._llcrypt unregistered [ 8939.490506] Key type ._llcrypt registered [ 8939.492761] Key type .llcrypt registered [ 8939.629866] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8956.318383] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8957.860923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8957.945114] alg: No test for adler32 (adler32-zlib) [ 8959.067612] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8959.334519] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8961.024217] Key type lgssc registered [ 8962.421788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8997.135786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8997.172769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.441933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8998.495990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8998.576869] Lustre: lustre-MDT0000: new disk, initializing [ 8998.723135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8998.739812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9003.658303] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9014.032815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.141421] Lustre: 298571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9014.189216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9014.194160] Lustre: Skipped 1 previous similar message [ 9014.323389] Lustre: lustre-MDT0001: new disk, initializing [ 9014.441305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9014.496533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9014.514436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9019.318067] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9030.903501] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9037.180129] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9044.797784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9045.076611] Lustre: lustre-OST0000: new disk, initializing [ 9045.094704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9045.114653] Lustre: 300540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9045.269700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.711505] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9053.739691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9053.766482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9053.867988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9061.987870] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9067.983285] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9081.212598] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9089.351648] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9096.217629] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9141.218505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9141.223689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9141.235743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.240836] Lustre: Skipped 1 previous similar message [ 9141.256252] Lustre: Skipped 1 previous similar message [ 9146.317143] Lustre: server umount lustre-OST0000 complete [ 9151.457890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.477155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9151.489209] Lustre: Skipped 1 previous similar message [ 9156.590892] Lustre: server umount lustre-MDT0000 complete [ 9160.016666] LustreError: 298562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453318 with bad export cookie 5989117461647192744 [ 9160.024657] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9160.031063] LustreError: 298562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9160.348882] Lustre: server umount lustre-MDT0001 complete [ 9168.860302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9173.926754] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9180.862560] Lustre: server umount MGS complete [ 9197.536394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.973789] LustreError: 304511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.999888] LustreError: 304511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9198.028480] LustreError: 304524:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9198.127550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9202.999610] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9203.192307] LustreError: 304512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.215666] LustreError: 304512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9207.279342] LustreError: 304511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.392905] LustreError: 304512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.497228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.878350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9218.768990] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9228.499620] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9237.489068] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9246.098574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9246.717847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.932538] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9262.859221] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.890670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9297.902618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.907404] LustreError: Skipped 1 previous similar message [ 9297.910158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9297.960138] Lustre: Skipped 2 previous similar messages [ 9302.646412] Lustre: server umount lustre-OST0000 complete [ 9308.137608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.148063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9308.161195] Lustre: Skipped 1 previous similar message [ 9308.175151] Lustre: Skipped 2 previous similar messages [ 9313.255256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.268062] Lustre: Skipped 1 previous similar message [ 9313.377467] Lustre: server umount lustre-MDT0000 complete [ 9317.348319] LustreError: 304512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9317.372535] LustreError: 304512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9317.714679] LustreError: 304491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453476 with bad export cookie 5989117461647194683 [ 9317.721839] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.727048] LustreError: 304491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9318.239113] Lustre: server umount lustre-MDT0001 complete [ 9329.304547] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 9332.367155] Key type lgssc unregistered [ 9332.696691] LNet: 308465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9332.703697] LNetError: 308465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9332.719237] LNet: Removed LNI 192.168.203.114@tcp [ 9333.816384] Key type .llcrypt unregistered [ 9333.821061] Key type ._llcrypt unregistered [ 9357.896409] Key type ._llcrypt registered [ 9357.900724] Key type .llcrypt registered [ 9358.129933] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 9374.025549] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9375.708936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9375.729257] alg: No test for adler32 (adler32-zlib) [ 9376.753592] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9376.992648] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 9378.672281] Key type lgssc registered [ 9379.845790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9403.313592] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9419.273347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9419.298413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.639608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9420.675515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9420.771522] Lustre: lustre-MDT0000: new disk, initializing [ 9420.877452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9420.901117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9425.649411] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9436.441441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9436.791797] Lustre: lustre-OST0000: new disk, initializing [ 9436.794506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9436.797894] Lustre: Skipped 1 previous similar message [ 9436.802885] Lustre: 312516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9436.814055] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9436.883065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9442.333714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9442.344406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9442.381914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9443.143973] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10315.203619] Lustre: server umount lustre-MDT0000 complete [10319.236029] LustreError: 311700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454478 with bad export cookie 10771403398673150303 [10319.255140] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.771922] Lustre: server umount lustre-OST0000 complete [10356.948659] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [10369.516894] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10405.313637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10405.546963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10405.589351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10405.673367] Lustre: lustre-MDT0000: new disk, initializing [10405.773914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10405.795264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10410.050420] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10420.353115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.525964] Lustre: 316857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10420.570883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10420.574737] Lustre: Skipped 1 previous similar message [10420.674158] Lustre: lustre-MDT0001: new disk, initializing [10420.759360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10420.782068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10420.787641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10424.977263] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10434.405908] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.520147] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10448.857131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.277093] Lustre: lustre-OST0000: new disk, initializing [10449.298834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10449.312238] Lustre: 318825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10449.514538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10455.790964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10455.808834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10456.017515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10458.103909] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10468.570424] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10472.830550] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10473.102772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10478.250950] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10478.501123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10481.634246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10481.635122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10481.640922] LustreError: Skipped 1 previous similar message [10481.650694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10481.660425] Lustre: Skipped 1 previous similar message [10486.934816] Lustre: server umount lustre-OST0000 complete [10491.878794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.880362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10491.899600] Lustre: Skipped 1 previous similar message [10491.915742] Lustre: Skipped 2 previous similar messages [10496.994766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10497.015221] Lustre: Skipped 1 previous similar message [10497.410182] Lustre: server umount lustre-MDT0000 complete [10501.089576] LustreError: 316849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454660 with bad export cookie 10771403398676091115 [10501.095846] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10501.103649] LustreError: 316849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10501.460543] Lustre: server umount lustre-MDT0001 complete [10505.272093] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10507.228911] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:57:44 (1788454664) [10513.972435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.431576] LustreError: 321139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.453968] LustreError: 321139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10514.506705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.671837] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10519.520972] LustreError: 321140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.648379] LustreError: 321139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10528.046888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.532653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10532.610266] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10542.630521] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10551.135766] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10560.200453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10560.657414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10568.825163] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10577.811415] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10587.748243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10588.013209] Lustre: lustre-OST0001: new disk, initializing [10588.023072] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10588.040822] Lustre: 323974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10588.177974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10593.865489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10593.887065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10593.929862] LustreError: 324275:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10594.980788] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10595.008738] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10595.096118] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10596.078881] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10598.132195] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10598.143096] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10602.273714] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10602.287555] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10606.234653] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10606.481933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10610.602947] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10610.620265] LustreError: 324269:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [10612.761351] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b91c72d8800.ost_server_uuid 50 [10614.586783] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b91c72d8800.ost_server_uuid in FULL state after 0 sec [10620.879454] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b91c72d8800.ost_server_uuid 50 [10622.531651] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b91c72d8800.ost_server_uuid in FULL state after 0 sec [10623.928944] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10639.330322] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10639.334690] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10639.353825] LustreError: Skipped 1 previous similar message [10639.354494] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10639.393606] Lustre: Skipped 1 previous similar message [10643.825049] Lustre: server umount lustre-OST0001 complete [10645.490683] LustreError: 322959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10645.518134] LustreError: 322959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10650.595063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10650.598159] LustreError: 324920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.598476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10650.598480] Lustre: Skipped 4 previous similar messages [10650.625067] Lustre: Skipped 1 previous similar message [10650.650210] LustreError: 324920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10656.593591] Lustre: server umount lustre-OST0000 complete [10660.833321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.837878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10660.846835] Lustre: Skipped 1 previous similar message [10660.865389] Lustre: Skipped 4 previous similar messages [10675.168738] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10675.533402] Lustre: server umount lustre-MDT0000 complete [10676.194596] LustreError: 321134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10676.220436] LustreError: 321134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10679.413183] LustreError: 321873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454838 with bad export cookie 10771403398676092004 [10679.420132] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10679.432590] LustreError: 321873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10679.874496] Lustre: server umount lustre-MDT0001 complete [10691.742928] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [10695.180039] Key type lgssc unregistered [10695.611875] LNet: 326666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10695.621287] LNetError: 326666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10695.636617] LNet: Removed LNI 192.168.203.114@tcp [10697.348906] Key type .llcrypt unregistered [10697.366606] Key type ._llcrypt unregistered [10728.044128] Key type ._llcrypt registered [10728.046200] Key type .llcrypt registered [10728.139235] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10730.146981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10730.174720] alg: No test for adler32 (adler32-zlib) [10731.442916] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10731.766987] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [10733.448190] Key type lgssc registered [10734.994578] Lustre: Echo OBD driver; http://www.lustre.org/ [10747.219724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10747.261176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10749.185594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10756.237224] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10759.661490] LustreError: 328101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10759.689776] LustreError: 328101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10764.782787] LustreError: 328102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.222027] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10769.889374] LustreError: 328101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10774.257071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.817951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10779.876840] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10789.710259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10790.158578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10795.308350] LustreError: 329780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.340859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10796.928102] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10799.409388] LustreError: 330054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10799.454014] LustreError: 330054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10805.046170] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10809.832738] LustreError: 329782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.896253] LustreError: 329782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10813.715594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10814.047784] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10819.080185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10822.270433] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10831.485312] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10838.581914] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10838.892949] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10947.043172] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10947.049124] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10947.075090] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10947.086023] Lustre: Skipped 1 previous similar message [10952.195347] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10952.199266] Lustre: Skipped 1 previous similar message [10957.281661] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10957.284516] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10957.317654] Lustre: Skipped 1 previous similar message [10957.421941] Lustre: server umount lustre-OST0001 complete [10962.402770] LustreError: 329781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.436380] LustreError: 329781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10964.448860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10964.468176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10964.490964] Lustre: Skipped 1 previous similar message [10964.504260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10967.536790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10972.657717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10972.673027] Lustre: Skipped 3 previous similar messages [10978.273579] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10978.679214] Lustre: server umount lustre-OST0000 complete [10984.934121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10984.956166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10984.991366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10985.002432] Lustre: Skipped 2 previous similar messages [10998.245370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10998.631989] Lustre: server umount lustre-MDT0000 complete [11003.363296] LustreError: 328098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.399038] LustreError: 328098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11003.881657] LustreError: 328082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455162 with bad export cookie 9367714243096548439 [11003.884574] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11003.917439] LustreError: 328082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11004.383685] Lustre: server umount lustre-MDT0001 complete [11018.338261] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11022.044835] Key type lgssc unregistered [11022.422910] LNet: 333053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11022.438245] LNetError: 333053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11023.464544] LNet: Removed LNI 192.168.203.114@tcp [11024.631340] Key type .llcrypt unregistered [11024.634876] Key type ._llcrypt unregistered [11077.643464] Key type ._llcrypt registered [11077.650415] Key type .llcrypt registered [11077.825771] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11079.913138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11079.938274] alg: No test for adler32 (adler32-zlib) [11081.165257] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11081.458312] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [11083.312186] Key type lgssc registered [11084.800960] Lustre: Echo OBD driver; http://www.lustre.org/ [11097.553728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11097.585557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.809683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11106.964127] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11109.361923] LustreError: 335345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.415433] LustreError: 335345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11114.466526] LustreError: 335346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.590128] LustreError: 335345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.928773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.818640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11127.505076] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11131.273729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11139.258803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11139.671887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11144.815614] LustreError: 336839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.851055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11146.476769] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11152.364464] LustreError: 336840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11152.375962] LustreError: 336840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11157.752797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.232822] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11163.661275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11166.714264] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11177.431608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11189.568467] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11204.582060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.586726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11204.606553] Lustre: Skipped 2 previous similar messages [11204.614792] Lustre: Skipped 3 previous similar messages [11209.355909] Lustre: server umount lustre-MDT0000 complete [11209.718756] LustreError: 336448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.750779] LustreError: 336448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11213.675847] LustreError: 336085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455372 with bad export cookie 9929278230378993541 [11213.686352] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11213.696765] LustreError: 336085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11214.017136] Lustre: server umount lustre-MDT0001 complete [11228.921959] Lustre: server umount lustre-OST0000 complete [11230.187480] Lustre: 334608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455373/real 1788455373] req@ffff924d39ab2680 x1875331241810560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11230.220340] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11230.243505] Lustre: Skipped 1 previous similar message [11233.331042] Lustre: server umount lustre-OST0001 complete [11253.071740] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11264.736987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11264.968307] LustreError: 340615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.016441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11270.725331] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11281.821797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11282.730056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11289.430267] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11293.037441] Lustre: 341755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11303.326113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11304.029139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11307.049166] LustreError: 342246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11307.076257] LustreError: 342246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11309.148644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11313.806577] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11323.790961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11329.025118] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11331.307894] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11339.547445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11343.505336] Lustre: 343632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11348.961257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11348.985563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11349.003219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11354.851316] Lustre: server umount lustre-MDT0000 complete [11358.833064] LustreError: 340596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455517 with bad export cookie 9929278230378995242 [11358.842918] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11358.860606] LustreError: 340596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11359.550470] Lustre: server umount lustre-MDT0001 complete [11374.550898] Lustre: server umount lustre-OST0000 complete [11375.457578] Lustre: 334609:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455518/real 1788455518] req@ffff924c1b270000 x1875331241863936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11375.512036] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.540673] Lustre: Skipped 3 previous similar messages [11379.038334] Lustre: server umount lustre-OST0001 complete [11384.038959] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11386.544597] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11389.476880] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 13:12:25 (1788455545) [11417.899652] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11431.543560] loop0: detected capacity change from 0 to 1099511627776 [11432.120148] SGI XFS with ACLs, security attributes, quota, no debug enabled [11432.127266] *** VALIDATE xfs *** [11432.136859] XFS (loop0): Mounting V5 Filesystem [11432.157884] XFS (loop0): Ending clean mount [11433.519875] loop1: detected capacity change from 0 to 18691697672192 [11569.735386] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [11584.798299] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11626.260941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.697925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11626.757259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11626.940132] Lustre: lustre-MDT0000: new disk, initializing [11627.090480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11627.100320] Lustre: Skipped 1 previous similar message [11627.154554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11632.416538] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11646.742293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.977627] Lustre: 348785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11647.153442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11647.162167] Lustre: Skipped 1 previous similar message [11647.374791] Lustre: lustre-MDT0001: new disk, initializing [11647.730588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11647.797974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11654.232919] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11667.744474] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11674.606824] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11682.622732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11682.916471] Lustre: lustre-OST0000: new disk, initializing [11682.920542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11682.934786] Lustre: 350760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11684.623615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11684.633891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11684.839071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11690.581155] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11701.628433] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11708.198627] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11708.544655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11713.492591] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11713.678172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11715.561516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11715.562362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.576862] LustreError: Skipped 1 previous similar message [11715.582357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.590675] Lustre: Skipped 1 previous similar message [11715.606568] Lustre: Skipped 5 previous similar messages [11720.689753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11720.704695] Lustre: Skipped 1 previous similar message [11722.065374] Lustre: server umount lustre-OST0000 complete [11729.380160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11729.394904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11729.418711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11741.152206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11741.155203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11741.177504] Lustre: Skipped 5 previous similar messages [11741.344873] Lustre: server umount lustre-MDT0000 complete [11745.252622] LustreError: 348796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11745.275669] LustreError: 348796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11746.048797] LustreError: 348776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455905 with bad export cookie 9929278230378996796 [11746.061237] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11746.065056] LustreError: 348776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11746.431192] Lustre: server umount lustre-MDT0001 complete [11754.942146] XFS (loop0): Unmounting Filesystem [11759.934675] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:18:36 (1788455916) [11767.513665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.028094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11768.039596] Lustre: Skipped 2 previous similar messages [11772.862799] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11782.422971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11787.750706] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11796.897663] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11802.435683] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11809.403623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11816.426592] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11824.982192] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11838.436503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11838.443821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.454932] Lustre: Skipped 1 previous similar message [11838.459418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11842.538956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11844.079613] Lustre: server umount lustre-OST0000 complete [11853.646372] Lustre: server umount lustre-MDT0000 complete [11857.881695] LustreError: 353302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456016 with bad export cookie 9929278230378997678 [11857.899807] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11858.412603] Lustre: server umount lustre-MDT0001 complete [11873.231195] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11877.077687] Key type lgssc unregistered [11877.539077] LNet: 356955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11877.553673] LNetError: 356955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11877.577326] LNet: Removed LNI 192.168.203.114@tcp [11878.944151] Key type .llcrypt unregistered [11878.946484] Key type ._llcrypt unregistered [11911.101771] Key type ._llcrypt registered [11911.106635] Key type .llcrypt registered [11911.194706] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11911.959827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11912.032554] alg: No test for adler32 (adler32-zlib) [11913.200507] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11913.357317] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [11914.992744] Key type lgssc registered [11916.218139] Lustre: Echo OBD driver; http://www.lustre.org/ [11924.836392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11924.883638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.766653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11934.043947] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11936.227423] LustreError: 358525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11936.245862] LustreError: 358525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11941.347673] LustreError: 358526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11945.542762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.833561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11950.703590] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11961.203595] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11969.055308] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11977.260577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11977.703069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11984.800806] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11993.093780] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12007.396122] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12007.398357] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.437754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12007.451017] Lustre: Skipped 1 previous similar message [12010.683673] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [12012.245081] Lustre: server umount lustre-MDT0001 complete [12012.521614] LustreError: 360368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.545643] LustreError: 360368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12015.758121] LustreError: 358520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.883163] LustreError: 360368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.919168] LustreError: 360368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12023.784845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12023.937925] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12023.950479] Lustre: Found index 1 for lustre-MDT0001, updating log [12024.013713] Lustre: 358512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12024.396473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12029.466310] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12029.472418] LustreError: 361955:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12029.513214] LustreError: 361955:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12029.552476] LustreError: 358517:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12029.684422] LustreError: 361972:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -17 [12029.710693] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12029.710693] [12029.793421] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12041.414632] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [12044.770678] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12044.778818] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.787046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12044.811930] Lustre: Skipped 2 previous similar messages [12046.907387] Lustre: server umount lustre-MDT0001 complete [12049.891401] LustreError: 359255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12049.917491] LustreError: 359255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12059.040307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12059.142780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12059.148719] Lustre: Found index 1 for lustre-MDT0001, updating log [12059.237120] Lustre: 358515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12059.702585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12064.755123] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12065.062802] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12065.857887] LustreError: 363080:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12065.874962] LustreError: 363080:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12065.886239] LustreError: 358517:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12065.899546] LustreError: 358517:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12065.985165] LustreError: 363088:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12065.992515] LustreError: 363088:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12065.999538] LustreError: 363088:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -17 [12066.026333] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.114@tcp [12066.026333] [12374.214746] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [12374.224743] Lustre: Skipped 2 previous similar messages [12377.063278] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.086078] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12380.155713] Lustre: server umount lustre-MDT0001 complete [12382.181830] LustreError: 361020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12382.249051] LustreError: 361020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12394.418613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.531614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12394.539479] Lustre: Found index 1 for lustre-MDT0001, updating log [12394.597632] Lustre: 358515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12395.177658] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12400.622360] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12400.643259] LustreError: 371085:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12400.655674] LustreError: 371085:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12400.666070] LustreError: 358517:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12400.671940] LustreError: 358517:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12400.721226] LustreError: 371088:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12400.743545] LustreError: 371088:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -114 [12400.753911] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12400.753911] [12401.235135] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12722.658105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12722.671473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.685620] Lustre: Skipped 1 previous similar message [12722.694979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12722.702725] Lustre: Skipped 3 previous similar messages [12729.032202] Lustre: server umount lustre-MDT0000 complete [12732.699027] LustreError: 358506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456891 with bad export cookie 10610733841123671505 [12732.711466] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12733.409732] LustreError: 358522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12733.415188] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12733.449277] LustreError: 358522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12733.505098] Lustre: Skipped 2 previous similar messages [12742.575844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12742.590687] Lustre: Skipped 8 previous similar messages [12747.232983] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12747.601198] Lustre: server umount lustre-MDT0001 complete [12751.858576] Lustre: server umount lustre-OST0000 complete [12761.995954] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [12772.884609] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [12810.794520] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:36:07 (1788456967) [12818.299502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12818.645236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12818.687565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12818.795493] Lustre: lustre-MDT0000: new disk, initializing [12818.947545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12818.979484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12824.008684] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12834.598606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12834.699623] Lustre: 381414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12834.742749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12834.746516] Lustre: Skipped 1 previous similar message [12834.801508] Lustre: lustre-MDT0001: new disk, initializing [12834.884926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12834.915355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12834.932315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12839.576328] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12849.929102] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12855.882360] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12862.616666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12863.017545] Lustre: lustre-OST0000: new disk, initializing [12863.020663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12863.025692] Lustre: 383380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12863.106694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12869.908326] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12870.820712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12870.852167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12871.009341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12880.404454] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12889.073203] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12889.088050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12889.115745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12890.130583] Lustre: server umount lustre-MDT0000 complete [12891.618649] LustreError: 382572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12891.645227] LustreError: 382572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12893.950344] LustreError: 381405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457052 with bad export cookie 10610733841123673451 [12893.956877] LustreError: 382220:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12893.963684] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12893.969194] LustreError: 381405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12894.384564] Lustre: server umount lustre-MDT0001 complete [12908.564319] Lustre: server umount lustre-OST0000 complete [12941.779166] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [12952.794608] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [12990.382417] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12670 sec ========= 13:39:06 (1788457146) [12992.038707] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:39:09 (1788457149) === [12996.041201] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:39:12 (1788457152) === [13037.673126] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [13040.642934] Key type lgssc unregistered [13040.918953] LNet: 389754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13040.932476] LNetError: 389754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13040.967690] LNet: Removed LNI 192.168.203.114@tcp [13042.060723] Key type .llcrypt unregistered [13042.063368] Key type ._llcrypt unregistered