[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 537118966 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002390] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.007031] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009165] LSM: Security Framework initializing [ 0.010067] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.013011] *** VALIDATE selinux *** [ 0.021610] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026878] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030172] *** VALIDATE tmpfs *** [ 0.031505] *** VALIDATE proc *** [ 0.032262] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035073] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036196] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042492] debug: unmapping init [mem 0xffffffffa6a59000-0xffffffffa6a60fff] [ 0.044211] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045769] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050008] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053340] rcu: Hierarchical SRCU implementation. [ 0.055508] smp: Bringing up secondary CPUs ... [ 0.056582] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.060569] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063023] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.130020] node 0 deferred pages initialised in 64ms [ 0.133111] devtmpfs: initialized [ 0.135283] x86/mm: Memory block size: 128MB [ 0.138740] gcov: version magic: 0x41383552 [ 0.141336] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143388] pinctrl core: initialized pinctrl subsystem [ 0.144191] [ 0.144818] ************************************************************* [ 0.145013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146014] ** ** [ 0.147014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148017] ** ** [ 0.149014] ** This means that this kernel is built to expose internal ** [ 0.150017] ** IOMMU data structures, which may compromise security on ** [ 0.151016] ** your system. ** [ 0.152014] ** ** [ 0.153014] ** If you see this message and you are not debugging the ** [ 0.154015] ** kernel, report this immediately to your vendor! ** [ 0.155014] ** ** [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157019] ************************************************************* [ 0.158664] NET: Registered protocol family 16 [ 0.159298] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.160041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.163012] cpuidle: using governor menu [ 0.165096] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168529] PCI: Using configuration type 1 for base access [ 0.170126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187038] cryptd: max_cpu_qlen set to 1000 [ 0.188279] ACPI: Added _OSI(Module Device) [ 0.190016] ACPI: Added _OSI(Processor Device) [ 0.191014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198124] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203583] ACPI: Interpreter enabled [ 0.205125] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207012] ACPI: Using IOAPIC for interrupt routing [ 0.208281] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211342] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222302] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.224049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.235899] acpiphp: Slot [2] registered [ 0.236171] acpiphp: Slot [5] registered [ 0.237000] acpiphp: Slot [6] registered [ 0.239100] acpiphp: Slot [7] registered [ 0.240104] acpiphp: Slot [8] registered [ 0.241095] acpiphp: Slot [9] registered [ 0.243120] acpiphp: Slot [10] registered [ 0.244101] acpiphp: Slot [3] registered [ 0.245147] acpiphp: Slot [4] registered [ 0.246093] acpiphp: Slot [11] registered [ 0.247079] acpiphp: Slot [12] registered [ 0.249101] acpiphp: Slot [13] registered [ 0.250140] acpiphp: Slot [14] registered [ 0.251095] acpiphp: Slot [15] registered [ 0.252099] acpiphp: Slot [16] registered [ 0.253081] acpiphp: Slot [17] registered [ 0.255121] acpiphp: Slot [18] registered [ 0.256116] acpiphp: Slot [19] registered [ 0.257115] acpiphp: Slot [20] registered [ 0.259085] acpiphp: Slot [21] registered [ 0.260107] acpiphp: Slot [22] registered [ 0.261139] acpiphp: Slot [23] registered [ 0.263097] acpiphp: Slot [24] registered [ 0.265185] acpiphp: Slot [25] registered [ 0.267176] acpiphp: Slot [26] registered [ 0.269193] acpiphp: Slot [27] registered [ 0.271175] acpiphp: Slot [28] registered [ 0.272148] acpiphp: Slot [29] registered [ 0.274222] acpiphp: Slot [30] registered [ 0.276212] acpiphp: Slot [31] registered [ 0.278085] PCI host bridge to bus 0000:00 [ 0.279042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281051] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.284039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.287046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290058] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.293043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.295256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299048] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301506] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.311822] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.317071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328713] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331786] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336934] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344028] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357036] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368919] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.375023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.381017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.397025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.406801] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.426017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.457817] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.464021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.470022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.487025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.498972] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.506019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.515015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.539017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548817] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.567017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.602656] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.616015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.643392] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.647265] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651509] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.654400] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.656261] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662077] iommu: Default domain type: Passthrough [ 0.663465] SCSI subsystem initialized [ 0.665236] ACPI: bus type USB registered [ 0.667101] usbcore: registered new interface driver usbfs [ 0.669110] usbcore: registered new interface driver hub [ 0.671105] usbcore: registered new device driver usb [ 0.673219] pps_core: LinuxPPS API ver. 1 registered [ 0.675013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.679064] PTP clock support registered [ 0.681073] EDAC MC: Ver: 3.0.0 [ 0.683221] PCI: Using ACPI for IRQ routing [ 0.685962] NetLabel: Initializing [ 0.687016] NetLabel: domain hash size = 128 [ 0.689013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.691106] NetLabel: unlabeled traffic allowed by default [ 0.693145] vgaarb: loaded [ 0.695277] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.697020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.701437] clocksource: Switched to clocksource kvm-clock [ 0.811158] VFS: Disk quotas dquot_6.6.0 [ 0.812897] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.816406] *** VALIDATE ramfs *** [ 0.817879] *** VALIDATE hugetlbfs *** [ 0.819567] pnp: PnP ACPI init [ 0.822210] pnp: PnP ACPI: found 6 devices [ 0.837840] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.841642] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.844970] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.847770] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.850545] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.853185] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.857326] NET: Registered protocol family 2 [ 0.861468] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.866731] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.870582] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.876465] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.880452] TCP: Hash tables configured (established 65536 bind 65536) [ 0.883827] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.887353] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890780] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.893971] NET: Registered protocol family 1 [ 0.896559] RPC: Registered named UNIX socket transport module. [ 0.898905] RPC: Registered udp transport module. [ 0.900215] RPC: Registered tcp transport module. [ 0.902033] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.904344] NET: Registered protocol family 44 [ 0.906156] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.908111] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.911571] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.913817] PCI: CLS 0 bytes, default 64 [ 0.916826] Unpacking initramfs... [ 2.363733] debug: unmapping init [mem 0xffff8e00bcc54000-0xffff8e00bffbffff] [ 2.367202] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.369331] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.372341] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.868722] Initialise system trusted keyrings [ 2.870622] Key type blacklist registered [ 2.873262] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.883303] zbud: loaded [ 2.886874] *** VALIDATE nfs *** [ 2.888279] *** VALIDATE nfs4 *** [ 2.889892] pstore: using deflate compression [ 2.894274] Platform Keyring initialized [ 3.005504] NET: Registered protocol family 38 [ 3.007446] Key type asymmetric registered [ 3.009175] Asymmetric key parser 'x509' registered [ 3.011535] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.015782] io scheduler mq-deadline registered [ 3.017328] io scheduler kyber registered [ 3.019162] io scheduler bfq registered [ 3.021134] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.027266] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.034715] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.039276] ACPI: Power Button [PWRF] [ 3.047921] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.068285] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.146730] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.162050] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.177792] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.207765] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.235534] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.240791] Non-volatile memory driver v1.3 [ 3.242306] Linux agpgart interface v0.103 [ 3.283554] virtio_blk virtio1: [vda] 136536 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.288267] vda: detected capacity change from 0 to 69906432 [ 3.335332] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.340429] vdb: detected capacity change from 0 to 1073741824 [ 3.380911] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.388315] vdc: detected capacity change from 0 to 2621440000 [ 3.416757] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.419493] vdd: detected capacity change from 0 to 2621440000 [ 3.458654] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.464692] vde: detected capacity change from 0 to 4294967296 [ 3.498753] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.501732] vdf: detected capacity change from 0 to 4294967296 [ 3.526217] libphy: Fixed MDIO Bus: probed [ 3.539429] usbcore: registered new interface driver usbserial_generic [ 3.547681] usbserial: USB Serial support registered for generic [ 3.549920] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.561459] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.565761] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.570933] mousedev: PS/2 mouse device common for all mice [ 3.576800] rtc_cmos 00:05: RTC can wake from S4 [ 3.585676] rtc_cmos 00:05: registered as rtc0 [ 3.587469] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.588586] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.595217] intel_pstate: CPU model not supported [ 3.598057] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.600950] hid: raw HID events driver (C) Jiri Kosina [ 3.606517] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.615640] usbcore: registered new interface driver usbhid [ 3.622367] usbhid: USB HID core driver [ 3.624976] drop_monitor: Initializing network drop monitor service [ 3.630632] Initializing XFRM netlink socket [ 3.636621] NET: Registered protocol family 10 [ 3.647483] Segment Routing with IPv6 [ 3.649193] NET: Registered protocol family 17 [ 3.659990] mpls_gso: MPLS GSO support [ 3.672937] RAS: Correctable Errors collector initialized. [ 3.683423] AVX version of gcm_enc/dec engaged. [ 3.687400] AES CTR mode by8 optimization enabled [ 3.927726] sched_clock: Marking stable (3927370232, 0)->(4873662447, -946292215) [ 3.935432] registered taskstats version 1 [ 3.946149] Loading compiled-in X.509 certificates [ 3.951896] zswap: loaded using pool lzo/zbud [ 4.035451] Key type big_key registered [ 4.086985] Key type encrypted registered [ 4.089513] ima: No TPM chip found, activating TPM-bypass! [ 4.108808] ima: Allocated hash algorithm: sha1 [ 4.115493] ima: No architecture policies found [ 4.119494] evm: Initialising EVM extended attributes: [ 4.123333] evm: security.selinux [ 4.129411] evm: security.ima [ 4.131949] evm: security.capability [ 4.139320] evm: HMAC attrs: 0x1 [ 4.151823] rtc_cmos 00:05: setting system clock to 2026-06-12 01:18:13 UTC (1781227093) [ 4.164074] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 4.174982] debug: unmapping init [mem 0xffffffffa6782000-0xffffffffa6a58fff] [ 4.185246] Write protecting the kernel read-only data: 28672k [ 4.192616] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 4.197739] debug: unmapping init [mem 0xffffffffa5714000-0xffffffffa57fffff] [ 4.327398] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.348517] systemd[1]: Detected virtualization kvm. [ 4.354478] systemd[1]: Detected architecture x86-64. [ 4.359695] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.449218] systemd[1]: No hostname configured. [ 4.451680] systemd[1]: Set hostname to . [ 4.457656] random: systemd: uninitialized urandom read (16 bytes read) [ 4.461437] systemd[1]: Initializing machine ID from random generator. [ 4.678782] random: ln: uninitialized urandom read (6 bytes read) [ 5.026830] random: systemd: uninitialized urandom read (16 bytes read) [ 5.039723] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.090111] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.104484] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. [ 7.069419] random: fast init done Starting dracut pre-udev hook... [ 7.428650] device-mapper: uevent: version 1.0.3 [ 7.450443] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.954000] virtio_net virtio0 ens2: renamed from eth0 [ 10.586149] scsi host0: ata_piix [ 10.613427] scsi host1: ata_piix [ 10.619703] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.632806] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.657849] random: crng init done [ 15.659435] random: 7 urandom warning(s) missed due to ratelimiting [ 17.497437] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.274967] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.169910] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.270322] SELinux: Disabled at runtime. [ 25.424790] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.455641] systemd[1]: Detected virtualization kvm. [ 25.457634] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.762045] hrtimer: interrupt took 4012691 ns [ 28.102328] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.141970] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.160457] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.171784] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.180435] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.214166] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.310145] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ 28.724535] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.284121] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.226764] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.400785] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.820195] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.876501] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 36.768750] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.282701] NFS: Registering the id_resolver key type [ 37.284869] Key type id_resolver registered [ 37.286700] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 94.137378] libcfs: loading out-of-tree module taints kernel. [ 94.230951] Key type ._llcrypt registered [ 94.232774] Key type .llcrypt registered [ 94.336713] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 112.843060] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 114.698620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.719934] alg: No test for adler32 (adler32-zlib) [ 116.120104] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 117.164625] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 119.008271] Key type lgssc registered [ 121.393356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.283822] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.548742] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 193.771072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 193.829362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.231978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.263128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.355636] Lustre: lustre-MDT0000: new disk, initializing [ 195.461549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.484370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.375394] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.756227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.831768] Lustre: 6514:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.863352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.873192] Lustre: Skipped 1 previous similar message [ 213.017700] Lustre: lustre-MDT0001: new disk, initializing [ 213.101245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.163222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.183126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.418944] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.050511] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.689106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.995329] Lustre: lustre-OST0000: new disk, initializing [ 230.999677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.011407] Lustre: 8417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.094392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.691532] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.189850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.208795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.318342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.389265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.466796] Lustre: lustre-OST0001: new disk, initializing [ 251.470319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.474770] Lustre: 9473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.526112] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.739585] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.186904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.193685] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.281693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.767158] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.364838] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.164138] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.939451] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 294.952038] Lustre: DEBUG MARKER: Client: 2.17.53.67 [ 297.975037] Lustre: DEBUG MARKER: MDS: 2.17.53.67 [ 300.478724] Lustre: DEBUG MARKER: OSS: 2.17.53.67 [ 302.027770] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 21:23:09 EDT 2026 [ 319.228205] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 320.806734] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.402086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.411803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.416508] Lustre: Skipped 1 previous similar message [ 338.438908] Lustre: Skipped 3 previous similar messages [ 342.521699] Lustre: server umount lustre-MDT0000 complete [ 345.893581] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227435 with bad export cookie 18300166270847859156 [ 345.899058] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.910954] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.640747] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.640993] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.642460] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.667099] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 348.697824] Lustre: Skipped 3 previous similar messages [ 350.689422] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.689676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.706574] Lustre: Skipped 2 previous similar messages [ 352.238126] Lustre: server umount lustre-MDT0001 complete [ 355.866359] Lustre: server umount lustre-OST0000 complete [ 359.721336] Lustre: server umount lustre-OST0001 complete [ 366.222445] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 375.510873] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 408.681841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.028536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.101956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.267435] Lustre: lustre-MDT0000: new disk, initializing [ 409.412687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.429167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 413.417906] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 424.229894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.366717] Lustre: 14986:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 424.446166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 424.449906] Lustre: Skipped 1 previous similar message [ 424.535499] Lustre: lustre-MDT0001: new disk, initializing [ 424.679580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.706876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.719363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.258183] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 436.436846] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 439.988907] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 447.138656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.484137] Lustre: lustre-OST0000: new disk, initializing [ 447.490741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.497210] Lustre: 16729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.607647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.386981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.394573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.530468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.612435] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 461.962673] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 467.697976] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 468.004904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 473.112456] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.327854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.226229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.245695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.296123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.302815] Lustre: Skipped 1 previous similar message [ 481.960784] Lustre: server umount lustre-OST0000 complete [ 490.471944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.475560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.485936] Lustre: Skipped 2 previous similar messages [ 490.499032] Lustre: Skipped 1 previous similar message [ 499.686910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 499.696959] Lustre: Skipped 3 previous similar messages [ 501.216719] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 501.460197] Lustre: server umount lustre-MDT0000 complete [ 504.808735] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.828408] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 505.564681] LustreError: 14977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227594 with bad export cookie 18300166270847860689 [ 505.566397] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.576599] LustreError: 14977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 506.044597] Lustre: server umount lustre-MDT0001 complete [ 510.973795] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 21:26:38 (1781227598) [ 516.971891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.627247] LustreError: 18847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.742494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.209962] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 523.234348] LustreError: 18848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.522267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.614846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 539.475248] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 542.926960] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.652950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.029535] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 561.031409] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 569.860483] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 577.002216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.010964] LustreError: Skipped 1 previous similar message [ 577.015758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.035304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.961928] Lustre: server umount lustre-MDT0000 complete [ 584.170265] LustreError: 19558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.192409] LustreError: 19558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 587.383815] LustreError: 19566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227676 with bad export cookie 18300166270847861571 [ 587.386788] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.396299] LustreError: 19566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.784600] Lustre: server umount lustre-MDT0001 complete [ 600.916183] Lustre: server umount lustre-OST0000 complete [ 619.669072] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 628.176721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.767973] LustreError: 22871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.854119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.860822] Lustre: Skipped 2 previous similar messages [ 632.969436] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 640.515372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.847336] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 647.700672] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 655.369918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 661.900582] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 669.638134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.774920] Lustre: lustre-OST0001: new disk, initializing [ 669.779119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 669.784730] Lustre: 25136:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 671.708271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 671.718881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 671.818695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 675.212891] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 683.485897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 686.599539] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 699.370771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.387039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.397804] Lustre: Skipped 5 previous similar messages [ 699.416548] Lustre: Skipped 6 previous similar messages [ 704.701616] Lustre: server umount lustre-MDT0000 complete [ 708.277722] LustreError: 25884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227797 with bad export cookie 18300166270847862418 [ 708.286378] LustreError: 25884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 708.286568] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.638358] Lustre: server umount lustre-MDT0001 complete [ 722.499967] Lustre: server umount lustre-OST0000 complete [ 725.477529] Lustre: 3666:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227798/real 1781227798] req@ffff8e000579de00 x1867752098995584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 725.491498] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.877229] Lustre: server umount lustre-OST0001 complete [ 739.421205] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 748.092537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.548100] LustreError: 28122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 748.566703] LustreError: 28122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 752.225642] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 759.521931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.995996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 760.008560] Lustre: Skipped 4 previous similar messages [ 763.497493] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 765.989645] Lustre: 29236:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 771.470265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.702558] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 786.429111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 793.195985] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 801.243367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 805.017359] Lustre: 31081:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 812.530190] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.533774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 812.549426] Lustre: Skipped 3 previous similar messages [ 812.568944] Lustre: Skipped 7 previous similar messages [ 817.788840] Lustre: server umount lustre-MDT0000 complete [ 822.006564] LustreError: 28103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227911 with bad export cookie 18300166270847863713 [ 822.011024] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 822.022354] LustreError: 28103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 822.275676] Lustre: server umount lustre-MDT0001 complete [ 836.379596] Lustre: server umount lustre-OST0000 complete [ 839.138985] Lustre: 3665:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227912/real 1781227912] req@ffff8e0136e892c0 x1867752099045248/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 839.705318] Lustre: server umount lustre-OST0001 complete [ 847.447423] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:32:15 (1781227935) [ 874.026612] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 883.712166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.148121] LustreError: 33864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.164651] LustreError: 33864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 888.508494] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 897.574629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.126062] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 906.371432] Lustre: 34980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 913.595401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 919.877653] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 927.793450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 933.054362] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 939.946938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 942.918902] Lustre: 36822:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 949.222523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 949.235622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.248753] Lustre: Skipped 1 previous similar message [ 949.255126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.263088] Lustre: Skipped 4 previous similar messages [ 952.785440] Lustre: server umount lustre-MDT0000 complete [ 955.997346] LustreError: 36824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228045 with bad export cookie 18300166270847864819 [ 956.013093] LustreError: 36824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 956.017650] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.309802] Lustre: server umount lustre-MDT0001 complete [ 970.316246] Lustre: server umount lustre-OST0000 complete [ 984.234204] Lustre: server umount lustre-OST0001 complete [ 993.620024] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:34:41 (1781228081) [ 1010.091935] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1020.223041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.567712] LustreError: 38951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.586048] LustreError: 38951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1020.686634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.690775] Lustre: Skipped 6 previous similar messages [ 1023.964137] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1030.704178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.235207] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1038.226685] Lustre: 40064:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1045.643947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1053.096080] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1062.207177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.263369] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1075.581345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1078.887774] Lustre: 41910:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1079.756664] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1079.766996] Lustre: Skipped 1 previous similar message [ 1082.367231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1082.381113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.398462] Lustre: Skipped 3 previous similar messages [ 1089.034827] Lustre: server umount lustre-MDT0000 complete [ 1092.932664] LustreError: 38933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228182 with bad export cookie 18300166270847865918 [ 1092.934788] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.947556] LustreError: 38933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1099.455951] Lustre: server umount lustre-MDT0001 complete [ 1108.322909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.140570] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfdf74043d078d8ac [ 1119.150489] Lustre: MGC192.168.204.131@tcp: Connection restored to 0@lo (at 0@lo) [ 1122.493421] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1127.729583] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1129.539455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.505382] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1135.092918] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1135.100528] Lustre: Skipped 1 previous similar message [ 1138.724733] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1140.125805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1143.220583] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1144.843947] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1150.434273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1155.969750] Lustre: server umount lustre-MDT0000 complete [ 1159.373028] LustreError: 38933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228248 with bad export cookie 18300166270847867052 [ 1159.384896] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.394629] LustreError: 38933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1159.742932] Lustre: server umount lustre-MDT0001 complete [ 1173.510468] Lustre: server umount lustre-OST0000 complete [ 1186.720583] Lustre: server umount lustre-OST0001 complete [ 1193.847656] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:38:01 (1781228281) [ 1206.600879] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1215.055749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.728993] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1225.822179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.138550] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1232.386581] Lustre: 47391:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1237.501345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.355962] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1250.858990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.523351] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1263.966757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1268.354986] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1281.278929] Lustre: server umount lustre-MDT0000 complete [ 1282.529832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1282.535814] LustreError: 47314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.560466] LustreError: 47314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 1284.227264] LustreError: 47009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228373 with bad export cookie 18300166270847867843 [ 1284.236314] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.252788] LustreError: 47009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1284.438829] Lustre: server umount lustre-MDT0001 complete [ 1298.718285] Lustre: server umount lustre-OST0000 complete [ 1312.179691] Lustre: server umount lustre-OST0001 complete [ 1319.760226] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 21:40:07 (1781228407) [ 1321.800384] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1323.243159] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 21:40:11 (1781228411) [ 1332.283903] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1340.074769] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1360.918601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.214321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1361.303915] Lustre: lustre-MDT0000: new disk, initializing [ 1361.402497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1365.937926] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1373.971683] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1380.744135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.899779] Lustre: lustre-OST0000: new disk, initializing [ 1380.905276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1380.910184] Lustre: Skipped 1 previous similar message [ 1380.916390] Lustre: 53280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1382.755939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1382.775185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1382.813676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1387.652340] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1395.016223] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1396.936289] Lustre: DEBUG MARKER: create test files [ 1423.846149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.860226] Lustre: Skipped 13 previous similar messages [ 1423.871331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1423.879138] Lustre: Skipped 22 previous similar messages [ 1437.154680] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1437.500777] Lustre: server umount lustre-OST0000 complete [ 1441.355250] Lustre: server umount lustre-MDT0000 complete [ 1451.352934] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1454.094521] Key type lgssc unregistered [ 1454.435361] LNet: 54777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.438948] LNetError: 54777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.464403] LNet: Removed LNI 192.168.204.131@tcp [ 1455.348156] Key type .llcrypt unregistered [ 1455.351241] Key type ._llcrypt unregistered [ 1482.935859] Key type ._llcrypt registered [ 1482.937601] Key type .llcrypt registered [ 1483.184525] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1485.259214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1485.294447] alg: No test for adler32 (adler32-zlib) [ 1486.607865] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 1486.922852] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1488.608418] Key type lgssc registered [ 1489.798727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1500.160676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1500.191035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.171738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.274846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1514.772768] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1522.375614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1522.674079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.790568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1526.990634] Lustre: 57381:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1527.006440] Lustre: 57381:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1527.021647] Lustre: 57381:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1527.028173] Lustre: 57381:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1527.051409] Lustre: 57381:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1527.062356] Lustre: 57381:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1528.707395] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1534.726710] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1539.799708] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1544.771403] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1552.356087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.386917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1553.522387] Lustre: server umount lustre-OST0000 complete [ 1556.859718] Lustre: server umount lustre-MDT0000 complete [ 1567.926181] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1570.830092] Key type lgssc unregistered [ 1571.180678] LNet: 58858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.199345] LNetError: 58858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.227168] LNet: Removed LNI 192.168.204.131@tcp [ 1572.054372] Key type .llcrypt unregistered [ 1572.057994] Key type ._llcrypt unregistered [ 1601.812608] Key type ._llcrypt registered [ 1601.815341] Key type .llcrypt registered [ 1601.944187] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1602.963426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1602.973523] alg: No test for adler32 (adler32-zlib) [ 1604.020587] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 1604.209598] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1605.904418] Key type lgssc registered [ 1606.970668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1613.499525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1613.522775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.938689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.878396] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1624.361794] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1631.109220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1631.422376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.789777] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1640.634448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1640.711179] Lustre: 61556:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1640.733858] Lustre: 61556:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1640.745521] Lustre: 61556:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1640.763624] Lustre: 61556:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1640.781109] Lustre: 61556:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1640.799207] Lustre: 61556:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1643.112963] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1648.224974] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1687.011836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.012323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.043877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.890739] Lustre: server umount lustre-OST0000 complete [ 1695.568335] Lustre: server umount lustre-MDT0000 complete [ 1707.179428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1710.428105] Key type lgssc unregistered [ 1710.769976] LNet: 63013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.780070] LNetError: 63013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.850626] LNet: Removed LNI 192.168.204.131@tcp [ 1712.725140] Key type .llcrypt unregistered [ 1712.732426] Key type ._llcrypt unregistered [ 1731.870185] Key type ._llcrypt registered [ 1731.873508] Key type .llcrypt registered [ 1731.982702] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1752.927229] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1754.409092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.561213] alg: No test for adler32 (adler32-zlib) [ 1755.734603] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 1756.276897] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1758.081542] Key type lgssc registered [ 1759.389966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.411720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.433446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.723950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1798.791054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1798.936355] Lustre: lustre-MDT0000: new disk, initializing [ 1799.056936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.090989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1803.591872] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1814.979617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.130902] Lustre: 66133:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1815.267386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1815.272034] Lustre: Skipped 1 previous similar message [ 1815.385969] Lustre: lustre-MDT0001: new disk, initializing [ 1815.503926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.562482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1815.583755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1820.834277] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1828.261320] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1832.260641] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1839.688907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.101874] Lustre: lustre-OST0000: new disk, initializing [ 1840.113416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1840.122889] Lustre: 67873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1840.245799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.418648] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1850.466158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1850.483946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1850.645316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1854.914517] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1859.973889] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1860.241980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1864.962849] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1865.192742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1871.331170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1871.350484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.371831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.386941] Lustre: Skipped 1 previous similar message [ 1873.566412] Lustre: server umount lustre-OST0000 complete [ 1881.570532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.584629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.598658] Lustre: Skipped 2 previous similar messages [ 1881.611382] Lustre: Skipped 1 previous similar message [ 1884.209801] Lustre: server umount lustre-MDT0000 complete [ 1888.705826] LustreError: 66123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228978 with bad export cookie 17281498537040739828 [ 1888.711340] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.717817] LustreError: 66123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1889.156291] Lustre: server umount lustre-MDT0001 complete [ 1898.021909] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:49:45 (1781228985) [ 1909.703987] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1946.807683] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1957.571034] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1995.792607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.251102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1996.330909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1996.449768] Lustre: lustre-MDT0000: new disk, initializing [ 1996.624375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.650895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2002.176027] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2013.909402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.022592] Lustre: 73697:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2014.075504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2014.079486] Lustre: Skipped 1 previous similar message [ 2014.137925] Lustre: lustre-MDT0001: new disk, initializing [ 2014.279816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.346350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2014.369520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2019.602611] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2028.063422] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2032.225685] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2039.819301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2040.082822] Lustre: lustre-OST0000: new disk, initializing [ 2040.087096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2040.095454] Lustre: 75439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2040.240921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.046176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2046.062750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2046.236519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2046.933464] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2055.366149] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2059.813354] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2060.021018] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2063.975282] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2064.232992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2066.916836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2066.921615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.930672] LustreError: Skipped 2 previous similar messages [ 2066.931740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2066.951817] Lustre: Skipped 1 previous similar message [ 2066.973776] Lustre: Skipped 1 previous similar message [ 2072.055219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2072.062632] Lustre: Skipped 1 previous similar message [ 2072.464797] Lustre: server umount lustre-OST0000 complete [ 2077.155599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.167203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.172044] Lustre: Skipped 1 previous similar message [ 2077.185017] Lustre: Skipped 1 previous similar message [ 2083.477773] Lustre: server umount lustre-MDT0000 complete [ 2087.394488] LustreError: 73707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.416803] LustreError: 73707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2087.654368] LustreError: 75438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229177 with bad export cookie 17281498537040740710 [ 2087.658442] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.670062] LustreError: 75438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2088.061838] Lustre: server umount lustre-MDT0001 complete [ 2099.244194] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:53:06 (1781229186) [ 2107.313259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.984494] LustreError: 77705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.087663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.085959] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2113.517234] LustreError: 77706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.542134] LustreError: 77706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2118.627115] LustreError: 77705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.114536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.728570] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2134.628252] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.021661] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2145.273960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.677360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.688217] Lustre: Skipped 1 previous similar message [ 2152.558599] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2158.371773] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2163.944379] Lustre: 80036:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 01596a1d-850f-4408-acb7-d9c3fdd0b125 at adminstrative request [ 2168.305268] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.327068] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2178.539421] Lustre: MGS: Client 01596a1d-850f-4408-acb7-d9c3fdd0b125 (at 0@lo) reconnecting [ 2178.555621] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xefd4375baebf04d8 to 0xefd4375baebf0778 [ 2178.568629] Lustre: MGC192.168.204.131@tcp: Connection restored to 0@lo (at 0@lo) [ 2201.911276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.087506] Lustre: lustre-OST0001: new disk, initializing [ 2202.091799] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2202.096572] Lustre: 80426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2209.245619] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2211.455279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2211.467504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2211.573974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2217.804241] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2222.050206] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2222.061067] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.073399] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2222.076846] Lustre: Skipped 1 previous similar message [ 2222.106610] Lustre: Skipped 3 previous similar messages [ 2227.523258] Lustre: server umount lustre-OST0001 complete [ 2232.295859] LustreError: 79301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.297248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.325411] LustreError: 79301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2232.359331] Lustre: Skipped 1 previous similar message [ 2238.502451] Lustre: server umount lustre-OST0000 complete [ 2246.114551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2246.125244] LustreError: Skipped 1 previous similar message [ 2246.158410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.167801] Lustre: Skipped 6 previous similar messages [ 2248.967343] Lustre: server umount lustre-MDT0000 complete [ 2252.701968] LustreError: 77687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229342 with bad export cookie 17281498537040742264 [ 2252.711213] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.722506] LustreError: 77687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2253.300457] Lustre: server umount lustre-MDT0001 complete [ 2261.702801] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 21:55:49 (1781229349) [ 2263.244171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2265.413219] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 21:55:52 (1781229352) [ 2267.272492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2269.086552] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 21:55:56 (1781229356) [ 2270.681549] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2272.883338] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 21:56:00 (1781229360) [ 2274.434801] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2276.174044] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 21:56:03 (1781229363) [ 2277.499719] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2284.233098] loop: module loaded [ 2284.273447] loop0: detected capacity change from 0 to 4294967296 [ 2284.330676] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2286.075720] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2289.758745] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2294.211763] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2323.347483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2323.406399] LustreError: 84419:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2323.418766] LustreError: 84419:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2323.425107] LustreError: 84419:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2323.433310] LustreError: 84419:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2344.069737] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 2355.411836] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2393.372895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.589618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2393.638960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2393.748702] Lustre: lustre-MDT0000: new disk, initializing [ 2393.892426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.904993] Lustre: Skipped 1 previous similar message [ 2393.937959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2398.074455] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2409.774222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.890993] Lustre: 87255:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2410.009785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2410.022476] Lustre: Skipped 2 previous similar messages [ 2410.340974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2415.638415] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2424.029612] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2429.671843] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2440.738196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2441.178797] Lustre: 88996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2449.703484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2450.003050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2451.038579] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2460.648760] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2466.410983] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2466.891868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2473.609489] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2473.890710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2480.613576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.626863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2480.641311] Lustre: Skipped 2 previous similar messages [ 2480.644566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.671605] Lustre: Skipped 1 previous similar message [ 2491.362105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2491.696728] Lustre: server umount lustre-OST0000 complete [ 2498.022712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2502.857443] Lustre: server umount lustre-MDT0000 complete [ 2506.210638] LustreError: 88282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.240361] LustreError: 88282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2507.547701] LustreError: 87247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229596 with bad export cookie 17281498537040742880 [ 2507.556076] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2507.559313] LustreError: 87247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2508.071048] Lustre: server umount lustre-MDT0001 complete [ 2519.553679] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 22:00:06 (1781229606) [ 2528.082761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.563945] LustreError: 91263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.617442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2528.623205] Lustre: Skipped 2 previous similar messages [ 2533.399579] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2543.714485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.200129] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2555.520593] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2559.743645] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2567.235655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.061636] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2581.542946] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2587.019117] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2587.340445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2592.467943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2592.710694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2600.849040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.047818] Lustre: lustre-OST0001: new disk, initializing [ 2601.052083] Lustre: Skipped 2 previous similar messages [ 2601.056772] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2601.062984] Lustre: Skipped 3 previous similar messages [ 2601.069824] Lustre: 94273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2608.220939] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2608.766546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2608.782392] Lustre: Skipped 2 previous similar messages [ 2608.799997] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2608.904868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2616.749457] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2622.729076] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2623.142742] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2628.379502] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2628.600218] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2650.124226] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2678.389798] Lustre: Failing over lustre-MDT0000 [ 2680.805157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.810889] LustreError: 92721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.821716] Lustre: Skipped 5 previous similar messages [ 2680.825072] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.846861] LustreError: 92721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2680.891942] Lustre: server umount lustre-MDT0000 complete [ 2706.003889] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2706.006889] LDISKFS-fs (dm-0): recovery complete [ 2706.014176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.406539] LustreError: 64355:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000088c0353f x1867753817658752/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2706.893776] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2707.497485] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2707.499029] Lustre: 94281:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2712.060361] Lustre: 91976:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2712.081766] Lustre: 91976:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 1 previous similar message [ 2712.101685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2712.195605] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2712.198258] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2712.546661] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2712.889846] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2712.903080] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 1 previous similar message [ 2713.252809] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2713.275070] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 1 previous similar message [ 2713.289661] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2713.306741] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 7 previous similar messages [ 2713.945494] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2713.966686] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 2 previous similar messages [ 2714.294400] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2714.305995] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 2 previous similar messages [ 2715.587707] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2715.605049] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 6 previous similar messages [ 2716.238061] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2716.253020] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 6 previous similar messages [ 2716.568322] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2716.585551] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 6 previous similar messages [ 2719.777308] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2719.795274] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 12 previous similar messages [ 2720.460227] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2720.482408] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 12 previous similar messages [ 2720.808239] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2720.821577] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 12 previous similar messages [ 2727.875085] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2727.919054] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 24 previous similar messages [ 2728.571067] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2728.578332] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 24 previous similar messages [ 2728.896643] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2728.923852] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 24 previous similar messages [ 2744.145587] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2744.183663] Lustre: 96974:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 50 previous similar messages [ 2744.841241] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2744.850591] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 50 previous similar messages [ 2745.178540] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2745.199361] LustreError: 96974:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 50 previous similar messages [ 2767.216211] Lustre: 96974:0:(ldlm_lib.c:2378:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2767.244870] Lustre: 96974:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2767.264257] Lustre: 96974:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 27529938-73f9-4445-9773-ba9a13db17c7@192.168.204.31@tcp [ 2767.278968] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2767.287440] Lustre: 96974:0:(ldlm_lib.c:1898:abort_req_replay_queue()) @@@ aborted: req@00000000c1ecd43a x1867753795422720/t0(8589934768) o35->27529938-73f9-4445-9773-ba9a13db17c7@192.168.204.31@tcp:647/0 lens 392/0 e 0 to 0 dl 1781229867 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2767.348820] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2767.375472] Lustre: lustre-MDT0000: Denying connection for new client 27529938-73f9-4445-9773-ba9a13db17c7 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2767.411073] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2767.457861] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2767.460732] Lustre: Skipped 2 previous similar messages [ 2767.466063] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2767.576877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2767.579798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2786.798768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.807297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.813490] Lustre: Skipped 2 previous similar messages [ 2786.829094] Lustre: Skipped 8 previous similar messages [ 2791.515882] Lustre: server umount lustre-OST0000 complete [ 2791.912778] LustreError: 92920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.948235] LustreError: 92920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 2811.360777] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2811.668947] Lustre: server umount lustre-OST0001 complete [ 2830.304882] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2830.587413] Lustre: server umount lustre-MDT0000 complete [ 2834.640875] LustreError: 91244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229923 with bad export cookie 17281498537040794092 [ 2834.648440] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.656973] LustreError: 91244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2835.079031] Lustre: server umount lustre-MDT0001 complete [ 2845.109523] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 22:05:32 (1781229932) [ 2852.938068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.948403] Lustre: *** cfs_fail_loc=197, val=0*** [ 2868.640920] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2873.760480] LustreError: 98936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2878.939093] Lustre: *** cfs_fail_loc=197, val=0*** [ 2878.943433] Lustre: Skipped 65 previous similar messages [ 2899.488453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.498885] Lustre: Skipped 4 previous similar messages [ 2907.367878] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2914.278850] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2917.236378] Lustre: server umount lustre-OST0000 complete [ 2928.183355] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 22:06:55 (1781230015) [ 2950.368746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.806460] LustreError: 100450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.827249] LustreError: 100450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2955.692251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2964.622603] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2968.817036] LustreError: 100481:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2968.830168] LustreError: 100481:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2968.840217] LustreError: 100481:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2969.189438] Lustre: server umount lustre-MDT0000 complete [ 3006.825549] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 3018.037298] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3059.046406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.386569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3059.447307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3059.559613] Lustre: lustre-MDT0000: new disk, initializing [ 3059.691435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3064.611765] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3076.583067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.877879] Lustre: 104522:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3077.450960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3082.075631] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3090.618211] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3095.190147] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3103.851044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3104.238628] Lustre: 106263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3111.219153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3113.069291] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3122.167433] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3127.406710] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3127.626431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3132.854501] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3133.058880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3137.000468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3137.007500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.010930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3137.010945] Lustre: Skipped 12 previous similar messages [ 3137.018343] LustreError: Skipped 1 previous similar message [ 3137.060622] Lustre: Skipped 5 previous similar messages [ 3141.968173] Lustre: server umount lustre-OST0000 complete [ 3161.056356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3161.322652] Lustre: server umount lustre-MDT0000 complete [ 3165.503957] LustreError: 104512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230254 with bad export cookie 17281498537040804669 [ 3165.517760] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3165.530718] LustreError: 104512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3166.202512] Lustre: server umount lustre-MDT0001 complete [ 3176.730736] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 22:11:03 (1781230263) [ 3186.408516] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3189.587521] Key type lgssc unregistered [ 3189.918143] LNet: 108589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3189.929650] LNetError: 108589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3189.950903] LNet: Removed LNI 192.168.204.131@tcp [ 3191.026725] Key type .llcrypt unregistered [ 3191.032046] Key type ._llcrypt unregistered [ 3210.178186] Key type ._llcrypt registered [ 3210.180676] Key type .llcrypt registered [ 3210.280065] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 3227.563319] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3228.886871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3228.937249] alg: No test for adler32 (adler32-zlib) [ 3230.076869] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 3230.338952] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3232.089115] Key type lgssc registered [ 3233.541794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3277.158827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3277.205633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.537792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3278.578564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3278.694236] Lustre: lustre-MDT0000: new disk, initializing [ 3278.798370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.826915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3283.287358] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3296.254175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.660432] Lustre: lustre-OST0000: new disk, initializing [ 3296.669471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3296.676716] Lustre: Skipped 1 previous similar message [ 3296.681647] Lustre: 112715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3296.789640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.508061] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3305.086922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3305.098368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3305.161203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3316.411105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.626983] Lustre: lustre-OST0001: new disk, initializing [ 3316.634721] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3316.646626] Lustre: 113560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3316.741917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.056479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3323.077591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3323.122357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3323.874964] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3333.091376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.099419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.124586] Lustre: Skipped 1 previous similar message [ 3338.211987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.218277] Lustre: Skipped 2 previous similar messages [ 3339.069477] Lustre: server umount lustre-MDT0000 complete [ 3346.948922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3351.810145] LustreError: 111917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230441 with bad export cookie 9397283438850168498 [ 3351.814537] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.821724] LustreError: 111917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3352.068177] Lustre: server umount lustre-OST0000 complete [ 3357.100996] Lustre: server umount lustre-OST0001 complete [ 3368.519045] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 22:14:15 (1781230455) [ 3379.803551] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3406.041611] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 22:14:53 (1781230493) [ 3449.252879] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 3460.666495] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3503.718479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.159230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3504.208794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3504.347665] Lustre: lustre-MDT0000: new disk, initializing [ 3504.461830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.518864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3509.344493] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3520.517899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.650229] Lustre: 120242:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3520.739563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3520.743922] Lustre: Skipped 1 previous similar message [ 3520.835219] Lustre: lustre-MDT0001: new disk, initializing [ 3521.000741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3521.056127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3521.063391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3526.172784] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3534.303778] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3538.343216] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3545.731338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3546.081434] Lustre: lustre-OST0000: new disk, initializing [ 3546.085746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3546.092092] Lustre: 121985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3546.213379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.387927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3553.403776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3553.580664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3554.918503] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3562.720637] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3567.413292] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3567.649811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3572.679932] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3572.913632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3577.318169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.328735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3577.343160] Lustre: Skipped 1 previous similar message [ 3577.350878] Lustre: Skipped 1 previous similar message [ 3582.437932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3582.447160] Lustre: Skipped 1 previous similar message [ 3587.563772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3587.573385] Lustre: Skipped 1 previous similar message [ 3589.090408] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3589.249670] Lustre: server umount lustre-OST0000 complete [ 3597.798777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3597.805041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.816189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.838699] Lustre: Skipped 1 previous similar message [ 3597.867715] Lustre: Skipped 1 previous similar message [ 3599.356758] Lustre: server umount lustre-MDT0000 complete [ 3602.844301] LustreError: 121984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230692 with bad export cookie 9397283438850169723 [ 3602.847310] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.851020] LustreError: 121984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3603.241104] Lustre: server umount lustre-MDT0001 complete [ 3611.796322] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 22:18:19 (1781230699) [ 3618.741437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.299429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.376823] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3624.422952] LustreError: 124255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.454872] LustreError: 124255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3629.550346] LustreError: 124254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.664487] LustreError: 124255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.322079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.824438] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3647.197171] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3651.387832] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3659.147934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3659.600555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.607158] Lustre: Skipped 1 previous similar message [ 3666.160124] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3672.376655] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3676.594460] Lustre: *** cfs_fail_loc=159, val=0*** [ 3676.602141] Lustre: Skipped 1 previous similar message [ 3692.121774] Lustre: lustre-MDT0000: Client fc37757b-df69-4976-8145-18ba1f1f93a4 (at 192.168.204.31@tcp) reconnecting [ 3693.589623] Lustre: *** cfs_fail_loc=159, val=0*** [ 3693.594016] Lustre: Skipped 4 previous similar messages [ 3710.020258] Lustre: lustre-MDT0000: Client fc37757b-df69-4976-8145-18ba1f1f93a4 (at 192.168.204.31@tcp) reconnecting [ 3716.067393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.078095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3716.083453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.089822] Lustre: Skipped 1 previous similar message [ 3729.377667] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3729.647418] Lustre: server umount lustre-OST0000 complete [ 3736.547541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.571764] Lustre: Skipped 1 previous similar message [ 3740.430432] Lustre: server umount lustre-MDT0000 complete [ 3741.668813] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.693572] LustreError: 125721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3744.009358] LustreError: 126514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230833 with bad export cookie 9397283438850170605 [ 3744.012385] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.022600] LustreError: 126514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3744.373682] Lustre: server umount lustre-MDT0001 complete [ 3754.985421] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3758.280968] Key type lgssc unregistered [ 3758.607939] LNet: 127671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3758.620501] LNetError: 127671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3758.639910] LNet: Removed LNI 192.168.204.131@tcp [ 3759.647174] Key type .llcrypt unregistered [ 3759.651982] Key type ._llcrypt unregistered [ 3782.735931] Key type ._llcrypt registered [ 3782.738434] Key type .llcrypt registered [ 3782.826463] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3783.737328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3783.762636] alg: No test for adler32 (adler32-zlib) [ 3784.880903] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 3785.234326] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3786.928320] Key type lgssc registered [ 3787.960672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3796.183824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3796.239092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.758975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.031251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3808.236969] LustreError: 129074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.261510] LustreError: 129074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3812.374997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.876283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3817.691477] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3823.410923] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3827.378738] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3834.211768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3834.549262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3840.279321] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3842.747453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3846.137241] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3856.021196] Lustre: *** cfs_fail_loc=159, val=0*** [ 3872.342558] Lustre: lustre-MDT0000: Client dc877da9-0527-45bf-a2f8-b06c725dc680 (at 192.168.204.31@tcp) reconnecting [ 3875.046116] Lustre: *** cfs_fail_loc=159, val=0*** [ 3875.051167] Lustre: Skipped 2 previous similar messages [ 3891.306396] Lustre: lustre-MDT0001: Client dc877da9-0527-45bf-a2f8-b06c725dc680 (at 192.168.204.31@tcp) reconnecting [ 3892.700843] Lustre: *** cfs_fail_loc=159, val=0*** [ 3892.702836] Lustre: Skipped 1 previous similar message [ 3908.206688] Lustre: lustre-MDT0001: Client dc877da9-0527-45bf-a2f8-b06c725dc680 (at 192.168.204.31@tcp) reconnecting [ 3912.867907] Lustre: 131774:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3915.663843] Lustre: *** cfs_fail_loc=159, val=0*** [ 3915.666216] Lustre: Skipped 4 previous similar messages [ 3931.256967] Lustre: lustre-MDT0000: Client 6b475fe9-739c-40a8-bc87-f2b40ea27ef2 (at 192.168.204.31@tcp) reconnecting [ 3933.025779] Lustre: *** cfs_fail_loc=159, val=0*** [ 3933.036407] Lustre: Skipped 14 previous similar messages [ 3949.674158] Lustre: lustre-MDT0000: Client 6b475fe9-739c-40a8-bc87-f2b40ea27ef2 (at 192.168.204.31@tcp) reconnecting [ 3951.898820] Lustre: 132066:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3960.809689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.810382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.829130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3960.832712] Lustre: Skipped 1 previous similar message [ 3960.869733] LustreError: Skipped 8 previous similar messages [ 3964.308522] Lustre: server umount lustre-OST0000 complete [ 3971.042897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.045705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3971.072895] Lustre: Skipped 1 previous similar message [ 3971.086073] Lustre: Skipped 2 previous similar messages [ 3976.167769] LustreError: 129786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.206665] LustreError: 129786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3976.383050] Lustre: server umount lustre-MDT0000 complete [ 3980.360823] LustreError: 131673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231069 with bad export cookie 4704504095335482164 [ 3980.381501] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3980.387501] LustreError: 131673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3980.703842] Lustre: server umount lustre-MDT0001 complete [ 3991.750526] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3994.392188] Key type lgssc unregistered [ 3994.725787] LNet: 133101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3994.739447] LNetError: 133101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3994.751988] LNet: Removed LNI 192.168.204.131@tcp [ 3995.868513] Key type .llcrypt unregistered [ 3995.870868] Key type ._llcrypt unregistered [ 4019.935567] Key type ._llcrypt registered [ 4019.940769] Key type .llcrypt registered [ 4020.049428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4021.107418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4021.128481] alg: No test for adler32 (adler32-zlib) [ 4022.142360] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 4022.404473] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4024.115218] Key type lgssc registered [ 4025.292687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4033.591969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4033.615115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.142309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.792666] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4045.282152] LustreError: 134500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.304535] LustreError: 134500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4050.324352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4050.415214] LustreError: 134499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4050.968100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4054.808761] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4060.571872] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4065.034200] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4072.473815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.875830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.987460] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4078.244972] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4080.078404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4082.870762] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4088.585789] Lustre: 136740:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4092.191908] Lustre: 136789:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4093.994884] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4095.067454] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4099.441632] Lustre: 136936:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4100.476510] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4101.715471] Lustre: 137045:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4110.819423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4110.827584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.838291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.868535] Lustre: Skipped 1 previous similar message [ 4114.089510] Lustre: server umount lustre-OST0000 complete [ 4121.069176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.083262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.097711] Lustre: Skipped 2 previous similar messages [ 4124.444346] Lustre: server umount lustre-MDT0000 complete [ 4126.184645] LustreError: 134499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.216856] LustreError: 134499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4128.054791] LustreError: 134480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231217 with bad export cookie 4901964560291962927 [ 4128.073522] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.284501] Lustre: server umount lustre-MDT0001 complete [ 4138.673706] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4141.457585] Key type lgssc unregistered [ 4141.733408] LNet: 138078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4141.742804] LNetError: 138078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4141.754736] LNet: Removed LNI 192.168.204.131@tcp [ 4142.597112] Key type .llcrypt unregistered [ 4142.599230] Key type ._llcrypt unregistered [ 4172.376987] Key type ._llcrypt registered [ 4172.380364] Key type .llcrypt registered [ 4172.587509] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4174.316145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4174.438654] alg: No test for adler32 (adler32-zlib) [ 4175.822874] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 4176.266681] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4178.017705] Key type lgssc registered [ 4179.454488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4187.446925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4187.508365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4189.049205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.423303] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4199.405362] LustreError: 139478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.424024] LustreError: 139478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4202.341344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.881562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4208.482891] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4214.902945] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4219.341936] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4226.785275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.189196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.229213] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4233.649259] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4234.368795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4239.938372] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4245.684667] Lustre: *** cfs_fail_loc=159, val=0*** [ 4245.699525] Lustre: Skipped 2 previous similar messages [ 4261.992926] Lustre: lustre-MDT0000: Client 7aa082bf-bafc-4303-9d2f-657215a2fd9e (at 192.168.204.31@tcp) reconnecting [ 4268.011894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4268.015781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.031644] LustreError: Skipped 2 previous similar messages [ 4268.055711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.553919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.568806] LustreError: Skipped 1 previous similar message [ 4269.571220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.579299] LustreError: 141082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.714653] Lustre: server umount lustre-OST0000 complete [ 4274.658943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4274.673762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.703472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4280.419641] Lustre: server umount lustre-MDT0000 complete [ 4280.801662] LustreError: 141102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.821958] LustreError: 141102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4284.708514] LustreError: 139458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231374 with bad export cookie 8300984570336446908 [ 4284.715387] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.719501] LustreError: 139458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4285.166558] Lustre: server umount lustre-MDT0001 complete [ 4296.583622] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4299.961632] Key type lgssc unregistered [ 4300.285828] LNet: 142785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4300.291970] LNetError: 142785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4300.307526] LNet: Removed LNI 192.168.204.131@tcp [ 4301.057973] Key type .llcrypt unregistered [ 4301.059797] Key type ._llcrypt unregistered [ 4328.001144] Key type ._llcrypt registered [ 4328.004237] Key type .llcrypt registered [ 4328.162760] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4329.233978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4329.431194] alg: No test for adler32 (adler32-zlib) [ 4330.425480] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 4330.686184] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4332.410074] Key type lgssc registered [ 4333.618520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4341.904335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4341.925993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.531535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.221022] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4352.997782] LustreError: 144202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.016078] LustreError: 144202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4358.123414] LustreError: 144203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.098280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.528798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4364.072930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4370.581625] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4375.319694] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4384.033160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4384.195462] LustreError: 145804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.732405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4387.883081] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4387.994452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4393.017800] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4399.585969] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4422.186391] Lustre: lustre-MDT0001: evicting 74976440-93e7-4b76-97ec-6b23dd075b96 (at 192.168.204.31@tcp) by administrative request [ 4422.191497] Lustre: Skipped 1 previous similar message [ 4454.373497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.375755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.391396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.418368] Lustre: Skipped 1 previous similar message [ 4459.504917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4459.507857] Lustre: Skipped 1 previous similar message [ 4464.609311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4464.619036] Lustre: Skipped 1 previous similar message [ 4466.145449] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4466.344376] Lustre: server umount lustre-OST0000 complete [ 4472.290222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4472.300344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.324038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.851817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.060270] Lustre: server umount lustre-MDT0000 complete [ 4479.971965] LustreError: 144916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.007419] LustreError: 144916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4481.932458] LustreError: 144182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231571 with bad export cookie 17270691366066537984 [ 4481.937610] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4481.947313] LustreError: 144182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4482.339062] Lustre: server umount lustre-MDT0001 complete [ 4494.149823] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4497.293683] Key type lgssc unregistered [ 4497.603455] LNet: 147917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4497.617618] LNetError: 147917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4497.643029] LNet: Removed LNI 192.168.204.131@tcp [ 4498.544150] Key type .llcrypt unregistered [ 4498.547885] Key type ._llcrypt unregistered [ 4579.562982] Key type ._llcrypt registered [ 4579.564961] Key type .llcrypt registered [ 4579.702486] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4580.875147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4580.904666] alg: No test for adler32 (adler32-zlib) [ 4582.120655] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 4582.319140] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4584.056206] Key type lgssc registered [ 4585.360752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4594.796773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4594.811479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.596728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.043323] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4605.922121] LustreError: 150394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.979236] LustreError: 150394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4611.050183] LustreError: 150395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.892574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.378615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4617.251260] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4623.859174] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4628.371486] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4636.579954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.189362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.300355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4641.323270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4644.089506] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4650.514217] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4661.734362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.743678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4661.776960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4661.790065] Lustre: Skipped 1 previous similar message [ 4666.854214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.184263] Lustre: server umount lustre-OST0000 complete [ 4671.972986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.977549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4671.996067] Lustre: Skipped 2 previous similar messages [ 4672.003454] Lustre: Skipped 1 previous similar message [ 4677.090974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4677.104960] Lustre: Skipped 1 previous similar message [ 4677.893299] Lustre: server umount lustre-MDT0000 complete [ 4682.209279] LustreError: 150395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.236045] LustreError: 150395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4682.300306] LustreError: 150375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231771 with bad export cookie 4444739962905501591 [ 4682.317235] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4682.626037] Lustre: server umount lustre-MDT0001 complete [ 4692.965665] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4696.160673] Key type lgssc unregistered [ 4696.541810] LNet: 153581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4696.553152] LNetError: 153581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4696.568386] LNet: Removed LNI 192.168.204.131@tcp [ 4697.678184] Key type .llcrypt unregistered [ 4697.681225] Key type ._llcrypt unregistered [ 4724.476665] Key type ._llcrypt registered [ 4724.478304] Key type .llcrypt registered [ 4724.573765] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4725.599964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4725.764948] alg: No test for adler32 (adler32-zlib) [ 4726.966934] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 4727.298135] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4728.976558] Key type lgssc registered [ 4730.223540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4744.113211] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 22:37:11 (1781231831) [ 4758.905541] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4769.415668] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4810.736831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4810.756293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.141769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4812.194889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4812.339696] Lustre: lustre-MDT0000: new disk, initializing [ 4812.499638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4812.531241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4818.080424] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4831.793758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.931355] Lustre: 157794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4832.017695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4832.024827] Lustre: Skipped 1 previous similar message [ 4832.119544] Lustre: lustre-MDT0001: new disk, initializing [ 4832.292836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4832.403790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4832.427470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4837.619494] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4846.682756] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4850.956681] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4859.229428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.752864] Lustre: lustre-OST0000: new disk, initializing [ 4859.759106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4859.764870] Lustre: 159536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4859.844813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4867.212458] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4867.682607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4867.701727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4867.851304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4875.501251] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4888.547778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4888.549915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.557177] LustreError: Skipped 1 previous similar message [ 4888.569528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4892.499878] Lustre: server umount lustre-OST0000 complete [ 4898.790150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4898.794938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.811929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4898.824890] Lustre: Skipped 2 previous similar messages [ 4898.850101] Lustre: Skipped 2 previous similar messages [ 4903.313773] Lustre: server umount lustre-MDT0000 complete [ 4907.730336] LustreError: 159535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231997 with bad export cookie 17188851944256530908 [ 4907.731373] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4907.754977] LustreError: 159535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4908.176994] Lustre: server umount lustre-MDT0001 complete [ 4919.744893] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4922.680676] Key type lgssc unregistered [ 4922.938980] LNet: 161310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4922.956138] LNetError: 161310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4924.008943] LNet: Removed LNI 192.168.204.131@tcp [ 4924.968879] Key type .llcrypt unregistered [ 4924.974604] Key type ._llcrypt unregistered [ 4951.087307] Key type ._llcrypt registered [ 4951.091329] Key type .llcrypt registered [ 4951.202777] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4952.361558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4952.382528] alg: No test for adler32 (adler32-zlib) [ 4953.523854] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 4953.799670] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4955.480127] Key type lgssc registered [ 4956.752329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.432516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.452881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.909657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.992189] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4976.101921] LustreError: 162707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.118301] LustreError: 162707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4978.480945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.958653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4983.574434] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4989.247668] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4993.245619] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5000.847936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5001.196524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5007.835946] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5011.455864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5014.336840] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5021.473574] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5021.485943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.495080] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5021.500691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.515713] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5022.488365] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5022.499559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.515665] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5022.525016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.532886] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5023.603871] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5023.633875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.645682] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5023.652793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.663598] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5024.693039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.686141] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5025.693802] Lustre: Skipped 1 previous similar message [ 5025.732614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.746684] Lustre: Skipped 1 previous similar message [ 5025.753732] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5025.759166] Lustre: Skipped 1 previous similar message [ 5025.766038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.785542] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5025.796897] Lustre: Skipped 1 previous similar message [ 5026.847113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.044360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.278365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.543616] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5030.552768] Lustre: Skipped 3 previous similar messages [ 5030.570536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.578229] Lustre: Skipped 3 previous similar messages [ 5030.582102] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5030.591522] Lustre: Skipped 3 previous similar messages [ 5030.594901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.605883] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5030.611730] Lustre: Skipped 3 previous similar messages [ 5031.804319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.794140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.682345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.493619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.567178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.464453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.439559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.354280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.510946] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5039.515626] Lustre: Skipped 8 previous similar messages [ 5039.536990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.549946] Lustre: Skipped 8 previous similar messages [ 5039.555518] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5039.560888] Lustre: Skipped 8 previous similar messages [ 5039.565309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.577616] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5039.591803] Lustre: Skipped 8 previous similar messages [ 5040.696160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.030866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.351218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.547161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.526300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.567484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.400304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.381742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.317287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.261981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.188598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.055219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.049568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.894521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.838918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.743136] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5055.749443] Lustre: Skipped 15 previous similar messages [ 5055.771208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5055.783608] Lustre: Skipped 15 previous similar messages [ 5055.787971] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5055.792362] Lustre: Skipped 15 previous similar messages [ 5055.795926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.814933] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5055.817936] Lustre: Skipped 15 previous similar messages [ 5056.667319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.479746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.391979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.258324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.300219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.156491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.250748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.127500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.299744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.372241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.445253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.468435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.459495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.506235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.464540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.539048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.577256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.523489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.586826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.586837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.837724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.886836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.867734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.995902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.027812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.117154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.226439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.265549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.278124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.192689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.208554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.219948] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5088.226047] Lustre: Skipped 31 previous similar messages [ 5088.253661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.265488] Lustre: Skipped 31 previous similar messages [ 5088.268895] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5088.279168] Lustre: Skipped 31 previous similar messages [ 5088.287866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.296192] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5088.303973] Lustre: Skipped 31 previous similar messages [ 5089.633471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.864161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.019591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.165437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.264876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.223227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.093046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.965422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.048449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.963959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.813343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.684888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.620737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.737519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.669970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.598074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.606976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.517818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.561287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.351297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.261516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.155476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.100675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.107820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.412649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.701913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.815884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.984834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.988099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.124089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.069779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.071155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.118131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.061918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.225212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.208972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.258571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.210664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.266646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.231179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.303329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.296478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.359855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.502184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.541851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.723841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.872016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.939733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.959663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.009222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.106479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.175226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.156346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.166404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.336598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.255766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.237520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.205378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.222392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.097237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.148376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.175821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.178960] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5153.182612] Lustre: Skipped 62 previous similar messages [ 5153.211226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.224453] Lustre: Skipped 62 previous similar messages [ 5153.228142] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5153.231437] Lustre: Skipped 62 previous similar messages [ 5153.237918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.247677] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5153.263657] Lustre: Skipped 62 previous similar messages [ 5154.235443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.280439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.379380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.554059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.712107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.750447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.760160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.672434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.617642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.709171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.602323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.648464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.625625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.640580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.624683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.503309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.478081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.754874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.641204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.455877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.418304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.949862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.147586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.247741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.093885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.254765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.290889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.365798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.615023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.592969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.701062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.769400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.786147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.704126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.194110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.244946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.449040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.620124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.978868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.296634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.364235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.607798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.836846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.091890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.154439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.330247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.436022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.429891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.765773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.050250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.277582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.372668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.448426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.629335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.604881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.593606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.530467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.500675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.306705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.327329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.259041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.304848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.369616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.252135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.274457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.245552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.260782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.156899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.024650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.053103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.131454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.352178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.277932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.293497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.427848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.384816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.443232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.547694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.610289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.650308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.503206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.595235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.644273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.682796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.609564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.730886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.577400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.633595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.438358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.453649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.307103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.245241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.187694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.108981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.144991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.170364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.099727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.999237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.932488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.779796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.689085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.647737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.775142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.950180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.917311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.884535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.614085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.658960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.706604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.877652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.831818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.754080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.007363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.316459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.873291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.313856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.991837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.803685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.555986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.266699] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5282.275142] Lustre: Skipped 119 previous similar messages [ 5282.298149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.314180] Lustre: Skipped 119 previous similar messages [ 5282.320854] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5282.331694] Lustre: Skipped 119 previous similar messages [ 5282.339622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.354872] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5282.363356] Lustre: Skipped 119 previous similar messages [ 5283.538338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.014120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.633879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.320702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.990053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.659192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.071672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.400830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.660512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.848335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.078645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.458635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.664412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.010562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.556533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.496713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.579188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.778459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.086900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.318108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.570064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.814822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.049167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.105041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.017349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.992345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.213890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.272290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.513439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.756315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.733376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.038290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.340747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.798438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.013231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.947946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.929878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.891172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.942354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.852805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.978527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.872320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.976286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.002212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.863755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.676317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.691187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.626907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.633111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.663976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.590354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.662202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.061652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.413716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.773440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.991661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.950065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.920420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.924393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.907061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.784363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.789918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.701021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.862167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.848428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.755591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.765694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.008753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.319325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.364828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.419647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.503743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.642622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.871102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.904736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.157196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.498573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.616955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.688282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.764843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.959328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.110429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.089910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.149458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.358137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.804632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.791463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.814721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.920896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.737994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.719928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.744419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.689277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.487495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.394179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.635911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.626363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.528380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.408297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.259481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.997094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.003387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.029259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.947500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.063808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.027221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.862091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.979846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.664320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.588151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.609447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.587441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.463515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.487390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.279645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.619973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.633839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.810238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.613654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.901043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.971322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.042653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.056508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.144923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.243666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.088139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.923645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.647258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.523687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.313122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.435390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.348891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.257366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.010444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.083261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.917223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.007686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.966856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.133489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.195380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.382493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.462665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.636824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.734051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.772757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.790493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.066518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.145705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.145429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.215548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.115741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.350276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.196817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.165444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.085576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.003103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.968675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.988147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.893501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.861530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.913919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.921185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.019951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.960510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.085526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.003280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.860258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.785943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.766187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.763850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.666616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.623283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.574484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.497245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.532560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.314908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.327878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.249382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.274468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.199264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.199917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.209738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.171585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.073825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.247377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.197261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.210240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.095533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.999330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.821963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.789701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.588245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.724591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.456542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.381843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.348566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.341654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.228598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.057926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.898062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.709988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.505891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.613675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.575239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.651290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.590177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.754570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.745132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.016091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.080573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.004829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.066509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.041327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.095690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.255213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.304658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.389579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.624748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.761565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.707777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.707583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.746792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.837762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.672701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.710115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.554510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.648587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.807250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.865928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.949209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.953765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.064952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.962204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.042976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.008099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.926137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.953617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.952798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.889398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.807795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.792380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.077262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.205586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.302804] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5538.316990] Lustre: Skipped 243 previous similar messages [ 5538.335980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.346461] Lustre: Skipped 243 previous similar messages [ 5538.351750] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5538.356849] Lustre: Skipped 243 previous similar messages [ 5538.362571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.378025] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5538.382806] Lustre: Skipped 243 previous similar messages [ 5539.272589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.216606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.052277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.054566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.197584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.435949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.490520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.478092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.619692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.821949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.436290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.574733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.244839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.229045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.495054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.151107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.666993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.719943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.760456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.237071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.709585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.838018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.067484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.031460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.948087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.109351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.030120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.015058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.244620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.168740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.446235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.234902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.238096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.308275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.124085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.976395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.683322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.417237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.245022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.169497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.997294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.841754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.884267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.052608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.978781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.859372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.960770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.848258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.830542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.780283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.764877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.679271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.619686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.438549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.353643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.345154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.464505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.318496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.170846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.138284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.204084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.219855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.478903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.824154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.893371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.909610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.897859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.908911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.844117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.930841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.048624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.008946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.851125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.704601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.660737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.597253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.660996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.694818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.652566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.661425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.732095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.041739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.102562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.185286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.339241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.256663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.180868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.031681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.846090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.709604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.788277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.641317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.570804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.633453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.656960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.659288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.555684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.768567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.865468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.869709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.929601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.851431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.892781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.782155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.796062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.843320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.799591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.928876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.741095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.142911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.187601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.254184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.319415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.191428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.115522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.071248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.972827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.881453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.794514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.699938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.584607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.662387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.672735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.809308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.746570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.898536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.737328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.719230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.707910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.738743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.659931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.702147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.633193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.875259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.993190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.177795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.486463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.748504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.938221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.023116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.161857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.310070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.312689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.211028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.152202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.173517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.059454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.040653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.910746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.989812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.057188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.164417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.360070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.438126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.647687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.057451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.319451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.600496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.749067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.665389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.863758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.925540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.934089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.072355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.069025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.110821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.376183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.378761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.485343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.464373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.579831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.655720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.657788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.717429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.719088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.846452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.777492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.811611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.777641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.883850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.209552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.245282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.392547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.449903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.443752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.317392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.341294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.559886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.822082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.017975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.002866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.086492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.120649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.098553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.351788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.564220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.642088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.705675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.548979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.655073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.654467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.711411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.071553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.081506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.116820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.066849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.978226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.868869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.793419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.746442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.026781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.921551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.120593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.160192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.482976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.659441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.613191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.021242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.241103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.275972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.142908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.286835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.442841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.730977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.976121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.862348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.223933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.696611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.913630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.171695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.361465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.524465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.984027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.285678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.440859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.536852] LustreError: 162726:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5787.553382] LustreError: 162726:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5787.556312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.528811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.803531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.923803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.061743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.242305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.275841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.472294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.534431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.875660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.903583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.247089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.419860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.426522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.662084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.847040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.821777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.997802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.138481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.356099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.487630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.447038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.764387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.873276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.037837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.209915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.047447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.174272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.328037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.155254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.228179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.413079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.236434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.551086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.880894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.062974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.363432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.878743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.359818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.628324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.712938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.739789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.807374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.826641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.999550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.152261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.352239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.471070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.594087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.721108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.071325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.135896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.118637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.073277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.147385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.874544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.845418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.807370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.946851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.020367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.036350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.923333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.209150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.476166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.733309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.306219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.740633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.179813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.763578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.209924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.492399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.025703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.071785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.152479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.218247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.275871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.418794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.764737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.647160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.634386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.552362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.754909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.756303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.011756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.852440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.895750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.833648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.775531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.813415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.796560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.765269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.826766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.870474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.933654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.206640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.164996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.296588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.540851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.610515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.682062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.707747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.848661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.730535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.593597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.724992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.695597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.931598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.025871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.897266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.976056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.883887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.960290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.004034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.968346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.981691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.085194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.964270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.136341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.273211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.275561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.408635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.338184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.411438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.341107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.311793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.488903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.618270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.651373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.973971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.198086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.689636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.880620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.164876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.333655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.529757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.228743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.446975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.734862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.990698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.826742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.156785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.436822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.866759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.044973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.056326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.459363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.505692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.845899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.113914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.065673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.240619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.159872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.124898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.263643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.423566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.892823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.709428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.828395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.730779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.770090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.825748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.912059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.936660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.135785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.493381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.628768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.771154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.868803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.797540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.873469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.793701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.906254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.888105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.099275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.261709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.455774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.649676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.832812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.106383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.273302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.414749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.396317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.784323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.911710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.910284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.943448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.892744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.021644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.030677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.242844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.574857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.825491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.100313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.140660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.802676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.992471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.315858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.802765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.762981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.698916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.621582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.891678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.863425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.076459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.310930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.327321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.460217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.594916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.651561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.642269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.590916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.741988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.845826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.942596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.834076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.005186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.114933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.173765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.223384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.280916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.488122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.387335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.397812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.328450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.380323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.377395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.344525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.269412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.676567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.765502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.738160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.030530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.187920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.167439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.292238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.150851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.242738] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6051.248547] Lustre: Skipped 471 previous similar messages [ 6051.264467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.272690] Lustre: Skipped 471 previous similar messages [ 6051.278575] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6051.286517] Lustre: Skipped 471 previous similar messages [ 6051.290980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.307513] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6051.323676] Lustre: Skipped 471 previous similar messages [ 6052.422769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.412480] LustreError: 162726:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6053.430137] LustreError: 162726:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6053.437517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.248766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.233452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.299421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.286737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.285860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.299136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.369337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.523596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.508637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.695519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.471298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.464111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.379355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.407697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.253343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.160554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.191572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.075421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.156269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.886394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.917461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.851116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.027067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.057141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.079971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.932912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.945634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.871957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.888812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.000778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.084101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.348752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.274727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.283449] LustreError: 164812:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6531.307108] LustreError: 162726:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7226.341493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7226.347885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.361493] Lustre: Skipped 35 previous similar messages [ 7226.368365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7227.363836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7228.569654] Lustre: server umount lustre-OST0000 complete [ 7237.089339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7237.104228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7237.113734] Lustre: Skipped 1 previous similar message [ 7239.170585] Lustre: server umount lustre-MDT0000 complete [ 7242.723766] LustreError: 162707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7242.746819] LustreError: 162707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7243.095547] LustreError: 162689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234332 with bad export cookie 12589834440227947960 [ 7243.098290] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.114706] LustreError: 162689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7243.534087] Lustre: server umount lustre-MDT0001 complete [ 7256.158712] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 7259.323979] Key type lgssc unregistered [ 7259.674616] LNet: 217647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7259.685859] LNetError: 217647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7259.713696] LNet: Removed LNI 192.168.204.131@tcp [ 7260.962128] Key type .llcrypt unregistered [ 7260.964201] Key type ._llcrypt unregistered [ 7286.553467] Key type ._llcrypt registered [ 7286.558236] Key type .llcrypt registered [ 7286.693602] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7288.052520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7288.118023] alg: No test for adler32 (adler32-zlib) [ 7289.337996] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 7289.666326] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 7291.392654] Key type lgssc registered [ 7293.034696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7301.271862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7301.311709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.850262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.646219] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7313.382271] LustreError: 219048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.409399] LustreError: 219048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7315.954293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.345700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7320.189629] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7325.403286] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7329.199449] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7335.645699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7335.991281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.550357] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7346.755132] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7348.282075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7352.466970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7354.277690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7354.296890] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7354.305594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7354.313555] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7354.316412] LustreError: 219067:0:(osp_precreate.c:1264:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7356.385304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7356.389624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.419269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.468076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.481991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.324913] Lustre: server umount lustre-OST0000 complete [ 7374.328327] LustreError: 219067:0:(osp_precreate.c:1264:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7398.242317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.787405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.806298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7399.838333] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7408.391448] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7409.090681] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7409.132569] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7416.852152] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7419.337517] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7424.756818] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b46012ac000.ost_server_uuid 50 [ 7426.467356] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b46012ac000.ost_server_uuid in IDLE state after 0 sec [ 7434.721561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7434.733183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7434.733761] Lustre: Skipped 1 previous similar message [ 7434.752102] Lustre: Skipped 3 previous similar messages [ 7439.334410] Lustre: server umount lustre-OST0000 complete [ 7444.964182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.992088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7449.570257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7449.584153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.904706] Lustre: server umount lustre-MDT0000 complete [ 7454.865563] LustreError: 219766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234544 with bad export cookie 14337542216849590328 [ 7454.875119] LustreError: 219766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7454.875277] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.482642] Lustre: server umount lustre-MDT0001 complete [ 7467.104552] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 7469.925812] Key type lgssc unregistered [ 7470.256836] LNet: 223696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7470.265509] LNetError: 223696:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7470.279195] LNet: Removed LNI 192.168.204.131@tcp [ 7471.105085] Key type .llcrypt unregistered [ 7471.107990] Key type ._llcrypt unregistered [ 7503.819278] Key type ._llcrypt registered [ 7503.827696] Key type .llcrypt registered [ 7503.928315] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7505.013616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7505.131302] alg: No test for adler32 (adler32-zlib) [ 7506.305472] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 7506.583247] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 7508.297368] Key type lgssc registered [ 7509.454348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7518.669065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7518.695273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.454690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.302351] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7529.955201] LustreError: 225408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.980841] LustreError: 225408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7533.103221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.396331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.913194] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7541.070445] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7548.975470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7549.564032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7556.874390] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7564.921781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7567.013075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7567.189841] Lustre: lustre-OST0001: new disk, initializing [ 7567.194359] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7567.199911] Lustre: 227665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7567.331502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.637412] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7576.228122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7576.245364] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7576.373387] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7585.785719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7591.927832] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7605.218240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7605.224962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.240480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.271231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.285382] Lustre: Skipped 2 previous similar messages [ 7607.285724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.303635] Lustre: Skipped 3 previous similar messages [ 7610.629451] Lustre: server umount lustre-MDT0000 complete [ 7612.395632] LustreError: 227943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.440394] LustreError: 227943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7615.226865] LustreError: 225389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234704 with bad export cookie 5469109821955717821 [ 7615.227655] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.235671] LustreError: 225389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7615.568662] Lustre: server umount lustre-MDT0001 complete [ 7630.171923] Lustre: server umount lustre-OST0000 complete [ 7632.865300] Lustre: 224670:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234706/real 1781234706] req@ffff8e0026cd4780 x1867759846779904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781234722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7632.894202] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.874545] Lustre: server umount lustre-OST0001 complete [ 7658.048361] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7671.828775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.421686] LustreError: 230741:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.525917] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.737341] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7677.943632] LustreError: 230742:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.032870] LustreError: 230741:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.966551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.463211] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7692.976238] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7695.870403] Lustre: 231852:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7702.996430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.159618] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7712.689069] LustreError: 232206:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7712.717135] LustreError: 232206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7716.803714] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 7720.367039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.799893] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7720.807370] Lustre: Skipped 1 previous similar message [ 7727.016642] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7734.310049] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7738.067896] Lustre: 233687:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7750.113695] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7750.121717] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.137261] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.781472] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.789387] Lustre: Skipped 7 previous similar messages [ 7761.888808] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7761.904116] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.908192] Lustre: Skipped 3 previous similar messages [ 7762.148590] Lustre: server umount mylustre-MDT0000 complete [ 7765.283721] LustreError: 230719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234854 with bad export cookie 5469109821955720628 [ 7765.285071] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7765.301414] LustreError: 230719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7765.706118] Lustre: server umount mylustre-MDT0001 complete [ 7779.621688] Lustre: server umount mylustre-OST0000 complete [ 7782.880234] Lustre: 224672:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234856/real 1781234856] req@ffff8e0026b01e00 x1867759846842240/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781234872 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7782.901587] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.915637] Lustre: Skipped 3 previous similar messages [ 7783.608147] Lustre: server umount mylustre-OST0001 complete [ 7804.863945] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7814.670244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.152425] LustreError: 235977:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.169422] LustreError: 235977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7815.255346] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.355256] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7827.562158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.893875] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7834.562524] Lustre: 237083:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7841.063460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7847.505212] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7853.668983] LustreError: 237437:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7853.686436] LustreError: 237437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7856.858907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.172073] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49792) [ 7860.184126] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7863.459925] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7871.400804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7875.549400] Lustre: 238920:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7884.264378] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7884.282612] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.306036] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7886.061721] Lustre: server umount tfs-MDT0000 complete [ 7890.466170] LustreError: 235954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234979 with bad export cookie 5469109821955722294 [ 7890.510649] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7891.141944] Lustre: server umount tfs-MDT0001 complete [ 7897.761509] Lustre: server umount tfs-OST0000 complete [ 7902.095464] Lustre: server umount tfs-OST0001 complete [ 7925.956667] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7935.754274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7936.101318] LustreError: 241152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7936.121824] LustreError: 241152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7936.217388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.223670] Lustre: Skipped 3 previous similar messages [ 7940.257308] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7947.925730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.776738] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7956.060173] Lustre: 242260:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7963.412751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7970.980772] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7979.875693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.119838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49824) [ 7981.160579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7987.140996] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7995.684411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7999.406653] Lustre: 244097:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8006.562376] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:31:34 (1781235094) [ 8025.060140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8025.078374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.092953] Lustre: Skipped 5 previous similar messages [ 8025.106910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8029.662479] Lustre: server umount lustre-MDT0000 complete [ 8033.899969] LustreError: 241128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235123 with bad export cookie 5469109821955723960 [ 8033.905392] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8033.909530] LustreError: 241128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8040.375459] Lustre: server umount lustre-MDT0001 complete [ 8058.339105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8058.415874] Lustre: server umount lustre-OST0000 complete [ 8076.256147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8076.457523] Lustre: server umount lustre-OST0001 complete [ 8083.941617] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8095.675140] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8133.568519] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8133.568519] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8133.568519] [ 8133.616613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8133.895052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8133.936813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8134.026570] Lustre: lustre-MDT0000: new disk, initializing [ 8134.104555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.110704] Lustre: Skipped 3 previous similar messages [ 8134.127220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8138.492304] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8150.799795] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8150.799795] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8150.799795] [ 8150.826833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8150.994645] Lustre: 247248:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8151.090241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8151.103114] Lustre: Skipped 1 previous similar message [ 8151.158407] Lustre: lustre-MDT0001: new disk, initializing [ 8151.320725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8151.334108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8156.188895] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8168.614945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.030486] Lustre: lustre-OST0000: new disk, initializing [ 8169.035890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8169.044707] Lustre: 248795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8170.175424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8170.201996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8170.471826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8176.731400] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8188.733941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8188.971192] Lustre: lustre-OST0001: new disk, initializing [ 8188.975343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8188.981303] Lustre: 249652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8190.666055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8190.674958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8190.791554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8195.637754] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8208.005924] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:34:55 (1781235295) [ 8224.898836] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:35:12 (1781235312) [ 8242.157095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.158428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.161450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.161461] Lustre: Skipped 8 previous similar messages [ 8242.170939] LustreError: Skipped 1 previous similar message [ 8242.208588] Lustre: Skipped 6 previous similar messages [ 8246.403328] Lustre: server umount lustre-OST0000 complete [ 8247.287858] LustreError: 248844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.313644] LustreError: 248844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 8257.035751] Lustre: server umount lustre-MDT0000 complete [ 8261.404591] LustreError: 247241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235350 with bad export cookie 5469109821955725514 [ 8261.410030] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8261.417635] LustreError: 247241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8268.127981] Lustre: server umount lustre-MDT0001 complete [ 8281.535352] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8293.649555] Lustre: server umount lustre-OST0001 complete [ 8300.655961] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8317.773862] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8359.483681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.769761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8359.832567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8359.999597] Lustre: lustre-MDT0000: new disk, initializing [ 8360.169417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8364.840256] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8375.868399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.967296] Lustre: 254139:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8376.020812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8376.024774] Lustre: Skipped 1 previous similar message [ 8376.091790] Lustre: lustre-MDT0001: new disk, initializing [ 8376.240156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8376.259196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8381.985966] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8391.858554] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8397.406601] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8405.923260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8406.297228] Lustre: 255882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8406.377848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.381990] Lustre: Skipped 5 previous similar messages [ 8407.877935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8408.001743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8412.284820] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8420.558277] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8428.524333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8428.529160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.539119] LustreError: Skipped 1 previous similar message [ 8428.540649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.548036] Lustre: Skipped 5 previous similar messages [ 8428.564501] Lustre: Skipped 7 previous similar messages [ 8433.921583] Lustre: server umount lustre-OST0000 complete [ 8444.212343] Lustre: server umount lustre-MDT0000 complete [ 8448.753185] LustreError: 254134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235538 with bad export cookie 5469109821955726984 [ 8448.757779] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.763993] LustreError: 254134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8449.139089] Lustre: server umount lustre-MDT0001 complete [ 8460.498267] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8463.885885] Key type lgssc unregistered [ 8464.258656] LNet: 257515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8464.284390] LNetError: 257515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8464.300507] LNet: Removed LNI 192.168.204.131@tcp [ 8465.151196] Key type .llcrypt unregistered [ 8465.159793] Key type ._llcrypt unregistered [ 8496.859483] Key type ._llcrypt registered [ 8496.862738] Key type .llcrypt registered [ 8497.013378] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8498.660178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8498.750401] alg: No test for adler32 (adler32-zlib) [ 8499.880466] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 8500.151091] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8501.960227] Key type lgssc registered [ 8503.574876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8513.565880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8513.616653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.255376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.215762] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8524.778581] LustreError: 258968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.829754] LustreError: 258968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8529.890270] LustreError: 258969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.506889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.034679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8535.675846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8542.977431] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8546.812485] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8553.698535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8554.154794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8560.117495] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8560.322741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8565.550630] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8566.606212] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8570.854914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.859049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.868590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.870270] Lustre: Skipped 1 previous similar message [ 8574.882470] Lustre: server umount lustre-OST0000 complete [ 8581.092720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.112701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.117241] Lustre: Skipped 1 previous similar message [ 8586.217324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.220710] Lustre: Skipped 1 previous similar message [ 8591.329038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8591.336788] Lustre: Skipped 2 previous similar messages [ 8593.376216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8593.516335] Lustre: server umount lustre-MDT0000 complete [ 8596.452081] LustreError: 258969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.463769] LustreError: 258969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8597.978454] LustreError: 261158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235687 with bad export cookie 8286197733094628384 [ 8597.981295] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8597.995909] LustreError: 261158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8598.563965] Lustre: server umount lustre-MDT0001 complete [ 8609.219645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8611.826734] Key type lgssc unregistered [ 8612.159498] LNet: 262198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8612.167514] LNetError: 262198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8612.192157] LNet: Removed LNI 192.168.204.131@tcp [ 8612.928673] Key type .llcrypt unregistered [ 8612.931819] Key type ._llcrypt unregistered [ 8630.060886] Key type ._llcrypt registered [ 8630.063810] Key type .llcrypt registered [ 8630.243437] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8631.181238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8631.339346] alg: No test for adler32 (adler32-zlib) [ 8632.410134] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 8632.639951] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8634.328461] Key type lgssc registered [ 8635.333730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8647.515475] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8647.585469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.392420] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8668.513427] LustreError: 263375:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 8695.201021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.249339] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8706.952121] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8710.194319] Lustre: server umount lustre-OST0000 complete [ 8728.455844] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8731.504537] Key type lgssc unregistered [ 8731.855667] LNet: 264877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8731.870291] LNetError: 264877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8732.904211] LNet: Removed LNI 192.168.204.131@tcp [ 8733.735059] Key type .llcrypt unregistered [ 8733.743603] Key type ._llcrypt unregistered [ 8775.128709] Key type ._llcrypt registered [ 8775.131103] Key type .llcrypt registered [ 8775.258955] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8776.614057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8776.658942] alg: No test for adler32 (adler32-zlib) [ 8777.734405] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 8778.043626] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8779.800228] Key type lgssc registered [ 8781.299878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8795.638796] loop0: detected capacity change from 0 to 204800000 [ 8795.888264] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8795.907133] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8796.283636] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8797.449401] loop0: detected capacity change from 0 to 204800000 [ 8797.626345] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8797.650277] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8798.113256] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8799.295905] loop0: detected capacity change from 0 to 204800000 [ 8799.495993] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8799.511171] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8799.721631] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8800.946871] loop0: detected capacity change from 0 to 204800000 [ 8801.135874] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8801.156314] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8801.283892] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8802.277828] loop0: detected capacity change from 0 to 204800000 [ 8802.323810] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8806.015741] loop0: detected capacity change from 0 to 204800000 [ 8806.042691] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8810.260449] loop0: detected capacity change from 0 to 204800000 [ 8810.299588] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8814.487745] loop0: detected capacity change from 0 to 204800000 [ 8814.536507] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8818.514431] loop0: detected capacity change from 0 to 204800000 [ 8818.619183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8818.693190] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8818.785841] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8818.793309] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8818.802395] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8818.811882] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8818.829981] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8818.837807] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8818.845778] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8818.855484] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8818.868820] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8818.883200] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8818.892187] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8818.901273] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8818.910451] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8818.924694] Lustre: 268274:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8824.161805] Lustre: server umount MGS complete [ 8825.164984] loop0: detected capacity change from 0 to 204800000 [ 8825.239748] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.873851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.917633] Lustre: lustre-MDT0000: Aborting client recovery [ 8826.783483] loop1: detected capacity change from 0 to 204800000 [ 8826.892844] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.975595] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8826.984187] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8827.006713] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8827.024195] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8827.042948] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8827.052254] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8827.065759] Lustre: 268707:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8827.090512] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8827.277540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8827.381814] Lustre: lustre-MDT0001: Aborting client recovery [ 8828.302882] loop2: detected capacity change from 0 to 204800000 [ 8828.386039] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8828.427073] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8828.453713] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8828.474178] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8828.491511] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8828.500265] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8828.509784] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8828.521693] Lustre: 268786:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8828.644867] Lustre: Found index 0 for lustre-OST0000, updating log [ 8828.683826] Lustre: 268803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8828.769072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8828.788055] Lustre: lustre-OST0000: Aborting recovery [ 8829.682199] loop3: detected capacity change from 0 to 204800000 [ 8829.787812] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.856494] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8829.870099] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8829.879315] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8829.892720] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8829.902712] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8829.918755] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8829.932096] Lustre: 268868:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8829.949831] Lustre: Found index 1 for lustre-OST0001, updating log [ 8829.994132] Lustre: 268874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8830.121167] Lustre: lustre-OST0001: Aborting recovery [ 8835.642499] LustreError: 268651:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8836.607517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8836.678452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8836.682475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8836.722662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8837.612356] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8837.612625] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8837.949315] LustreError: 268617:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 15074us: rc = -2 [ 8838.459963] LustreError: 268617:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 671us: rc = -2 [ 8838.488224] LustreError: 268617:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 23 previous similar messages [ 8852.960905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8852.965134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8852.974837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8853.013717] Lustre: Skipped 2 previous similar messages [ 8857.885968] Lustre: server umount lustre-MDT0000 complete [ 8858.097874] LustreError: 268620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8858.115623] LustreError: 268620:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8859.200113] LustreError: 268600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235948 with bad export cookie 5757208843328500719 [ 8859.203256] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8859.211564] LustreError: 268600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8863.201281] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.202560] LustreError: 268616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.226575] Lustre: Skipped 1 previous similar message [ 8863.227227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8863.227232] Lustre: Skipped 3 previous similar messages [ 8863.320321] LustreError: 268616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8864.228608] LustreError: 268882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8864.234461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8864.265673] LustreError: 268882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8864.320050] Lustre: Skipped 2 previous similar messages [ 8865.696187] Lustre: server umount lustre-MDT0001 complete [ 8873.147083] Lustre: server umount lustre-OST0000 complete [ 8888.288134] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8888.544656] Lustre: server umount lustre-OST0001 complete [ 8895.972512] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:46:23 (1781235983) [ 8924.416273] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8934.230815] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8968.910736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.170694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8969.215729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8969.322760] Lustre: lustre-MDT0000: new disk, initializing [ 8969.453533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.460055] Lustre: Skipped 1 previous similar message [ 8969.473553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8973.971153] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8985.146428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.308187] Lustre: 272909:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8985.412695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8985.421530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8985.433923] Lustre: Skipped 1 previous similar message [ 8985.529840] Lustre: lustre-MDT0001: new disk, initializing [ 8985.652153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8985.688783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8985.708395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8990.033431] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8997.714673] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9000.987844] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9007.566836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.049515] Lustre: lustre-OST0000: new disk, initializing [ 9008.065561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9008.085507] Lustre: 274650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9008.226318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.929745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9014.940963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9014.970049] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9015.058969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9022.638725] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9027.463226] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9039.499579] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9048.745563] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9058.908398] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9107.429904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.437158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.454111] Lustre: Skipped 2 previous similar messages [ 9107.456543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9107.481495] Lustre: Skipped 1 previous similar message [ 9111.129649] Lustre: server umount lustre-OST0000 complete [ 9117.667839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.675379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9117.710965] Lustre: Skipped 1 previous similar message [ 9122.758613] Lustre: server umount lustre-MDT0000 complete [ 9122.801230] LustreError: 272922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.819229] LustreError: 272922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9126.884137] LustreError: 272917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.909226] LustreError: 272917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9126.995232] LustreError: 274648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236216 with bad export cookie 5757208843328503435 [ 9127.000246] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9127.020952] LustreError: 274648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9127.596449] Lustre: server umount lustre-MDT0001 complete [ 9137.165426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.771800] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9147.074804] Lustre: server umount MGS complete [ 9157.872230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.340158] LustreError: 278537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.375206] LustreError: 278550:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9158.424656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9162.789966] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9170.816329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.035032] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9181.647030] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9185.182859] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9191.255245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.518997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.524451] Lustre: Skipped 1 previous similar message [ 9196.635257] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9201.713170] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9233.902613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9233.905319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.915407] LustreError: Skipped 2 previous similar messages [ 9233.919427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9233.926945] Lustre: Skipped 1 previous similar message [ 9233.943507] Lustre: Skipped 2 previous similar messages [ 9239.766188] Lustre: server umount lustre-OST0000 complete [ 9243.106221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9243.118628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.357582] Lustre: server umount lustre-MDT0000 complete [ 9252.989431] LustreError: 278519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236342 with bad export cookie 5757208843328505374 [ 9252.991179] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.002097] LustreError: 278519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9253.423756] Lustre: server umount lustre-MDT0001 complete [ 9264.069567] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9267.502540] Key type lgssc unregistered [ 9267.862731] LNet: 282127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9267.881755] LNetError: 282127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9267.935293] LNet: Removed LNI 192.168.204.131@tcp [ 9269.039059] Key type .llcrypt unregistered [ 9269.047843] Key type ._llcrypt unregistered [ 9312.471211] Key type ._llcrypt registered [ 9312.473524] Key type .llcrypt registered [ 9312.578532] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9330.168594] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9331.871090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9331.893724] alg: No test for adler32 (adler32-zlib) [ 9332.990471] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 9333.243900] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9334.968264] Key type lgssc registered [ 9336.328141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9375.673618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9375.716530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.033165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9377.125345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9377.233747] Lustre: lustre-MDT0000: new disk, initializing [ 9377.295334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.318299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9382.107414] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9392.713390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9392.826579] Lustre: 286060:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9392.894161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9392.902882] Lustre: Skipped 1 previous similar message [ 9392.997928] Lustre: lustre-MDT0001: new disk, initializing [ 9393.113813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9393.154511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9393.168198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9397.296436] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9404.189220] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9408.525847] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9417.448916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.784189] Lustre: lustre-OST0000: new disk, initializing [ 9417.790183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9417.807567] Lustre: 287801:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9417.970208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.561040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9424.568814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9424.712143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9425.073498] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9433.512486] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9441.162734] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9450.945401] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9458.823940] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9466.941181] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9506.802035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9506.806232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.834436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9506.839200] Lustre: Skipped 1 previous similar message [ 9511.908704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9511.922153] Lustre: Skipped 1 previous similar message [ 9517.031389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9517.042725] Lustre: Skipped 1 previous similar message [ 9520.097965] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9520.258779] Lustre: server umount lustre-OST0000 complete [ 9526.252778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9526.262276] LustreError: Skipped 1 previous similar message [ 9526.264221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.279853] Lustre: Skipped 1 previous similar message [ 9526.293923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9527.274161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.420991] Lustre: server umount lustre-MDT0000 complete [ 9534.900305] LustreError: 288544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236624 with bad export cookie 15102849396255802293 [ 9534.907721] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9534.917345] LustreError: 288544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9535.357457] Lustre: server umount lustre-MDT0001 complete [ 9548.925634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9553.975251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9560.329704] Lustre: server umount MGS complete [ 9571.862071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9572.376395] LustreError: 291542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.407794] LustreError: 291542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9572.413508] LustreError: 291555:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9572.467252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.711514] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9577.971989] LustreError: 291543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9582.059833] LustreError: 291542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9585.013465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.539634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9590.193689] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9596.271925] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9600.178781] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9607.495782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9607.790840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.265252] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9619.780657] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9651.177285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9651.194358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.217644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9651.223620] Lustre: Skipped 2 previous similar messages [ 9653.225269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9655.869346] Lustre: server umount lustre-OST0000 complete [ 9662.436267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9662.446970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9662.457861] Lustre: Skipped 1 previous similar message [ 9662.465184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9662.468042] Lustre: Skipped 2 previous similar messages [ 9666.838759] Lustre: server umount lustre-MDT0000 complete [ 9668.580228] LustreError: 293931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9668.612592] LustreError: 293931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9670.938573] LustreError: 291523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236760 with bad export cookie 15102849396255804246 [ 9670.942112] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.956860] LustreError: 291523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9671.409265] Lustre: server umount lustre-MDT0001 complete [ 9682.912948] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9686.773311] Key type lgssc unregistered [ 9687.138398] LNet: 295133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9687.145157] LNetError: 295133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9687.171466] LNet: Removed LNI 192.168.204.131@tcp [ 9688.225584] Key type .llcrypt unregistered [ 9688.229138] Key type ._llcrypt unregistered [ 9711.507637] Key type ._llcrypt registered [ 9711.511417] Key type .llcrypt registered [ 9711.640669] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9728.300347] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9729.527658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9729.578157] alg: No test for adler32 (adler32-zlib) [ 9730.682502] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [ 9730.898759] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9732.600511] Key type lgssc registered [ 9733.912947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9758.651630] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9772.189246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9772.239224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.661685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9773.712378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9773.849701] Lustre: lustre-MDT0000: new disk, initializing [ 9774.069259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.106262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9779.082227] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9791.383127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9791.669949] Lustre: lustre-OST0000: new disk, initializing [ 9791.679107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9791.684145] Lustre: Skipped 1 previous similar message [ 9791.691084] Lustre: 299133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9791.763906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.768735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9798.782852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9798.816079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9798.850625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10761.186137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.453785] Lustre: server umount lustre-MDT0000 complete [10765.974846] LustreError: 298323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237855 with bad export cookie 12450210356068514938 [10765.984433] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10765.985156] LustreError: 298323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10766.152443] Lustre: server umount lustre-OST0000 complete [10795.718971] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10807.632207] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10844.053075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.223458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10844.265541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10844.359448] Lustre: lustre-MDT0000: new disk, initializing [10844.449590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10844.462960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10849.826275] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10862.982028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10863.082381] Lustre: 303441:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10863.203600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10863.213302] Lustre: Skipped 1 previous similar message [10863.361782] Lustre: lustre-MDT0001: new disk, initializing [10863.526664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10863.592645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10863.604106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10869.159178] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10876.600763] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10881.670356] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10889.739842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10890.137713] Lustre: lustre-OST0000: new disk, initializing [10890.145975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10890.164517] Lustre: 305182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10890.316787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10897.384706] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10898.538962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10898.552964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10898.724700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10905.881844] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10911.251825] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10911.435210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10916.544984] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10916.839897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10919.398877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10919.399372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10919.416272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10919.429777] Lustre: Skipped 1 previous similar message [10919.468317] LustreError: Skipped 1 previous similar message [10924.517808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10924.524511] Lustre: Skipped 1 previous similar message [10925.250995] Lustre: server umount lustre-OST0000 complete [10929.634423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10929.636214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.656987] Lustre: Skipped 1 previous similar message [10929.667128] Lustre: Skipped 2 previous similar messages [10934.755434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10935.759955] Lustre: server umount lustre-MDT0000 complete [10939.873321] LustreError: 303452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.889487] LustreError: 303452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10940.382271] LustreError: 303431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238029 with bad export cookie 12450210356071455757 [10940.385609] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.396406] LustreError: 303431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10941.028441] Lustre: server umount lustre-MDT0001 complete [10945.401326] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10947.250988] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 00:20:34 (1781238034) [10954.981789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.481987] LustreError: 307348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.530144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10960.526743] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10960.869794] LustreError: 307349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10964.983041] LustreError: 307348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.973500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.092720] LustreError: 307349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.586664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10975.058519] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10981.699273] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10985.620542] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10993.147766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.449614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10999.747947] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11005.190471] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11013.051784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11013.299769] Lustre: lustre-OST0001: new disk, initializing [11013.307162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11013.312977] Lustre: 309843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11013.421149] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11020.673879] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11022.924306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11022.938553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11023.002262] LustreError: 310288:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11024.035372] LustreError: 310288:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11024.052523] LustreError: 310288:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11025.060274] LustreError: 310292:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11027.111253] LustreError: 310292:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11027.122224] LustreError: 310292:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11030.180723] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11030.478668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11031.270408] LustreError: 310288:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11031.289209] LustreError: 310288:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [11035.206249] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b4607b99000.ost_server_uuid 50 [11036.942281] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9b4607b99000.ost_server_uuid in FULL state after 0 sec [11039.462326] LustreError: 310292:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11039.491942] LustreError: 310292:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11040.977859] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b4607b99000.ost_server_uuid 50 [11042.592558] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b4607b99000.ost_server_uuid in FULL state after 0 sec [11044.734558] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11059.182104] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.192629] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11059.207966] Lustre: Skipped 1 previous similar message [11059.243699] Lustre: Skipped 2 previous similar messages [11064.088802] Lustre: server umount lustre-OST0001 complete [11064.300812] LustreError: 309502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11064.320356] LustreError: 309502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11072.483531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11072.484128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11072.493986] LustreError: Skipped 1 previous similar message [11072.495864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11072.510673] Lustre: Skipped 1 previous similar message [11072.523147] Lustre: Skipped 2 previous similar messages [11076.641774] Lustre: server umount lustre-OST0000 complete [11083.233719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11083.253067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11086.886740] Lustre: server umount lustre-MDT0000 complete [11089.890785] LustreError: 308329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11089.911602] LustreError: 308329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11090.440222] LustreError: 307328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238179 with bad export cookie 12450210356071456639 [11090.441770] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11090.452110] LustreError: 307328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11090.824737] Lustre: server umount lustre-MDT0001 complete [11103.070026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11105.855381] Key type lgssc unregistered [11106.229718] LNet: 312289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11106.240953] LNetError: 312289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11106.271402] LNet: Removed LNI 192.168.204.131@tcp [11107.273979] Key type .llcrypt unregistered [11107.276987] Key type ._llcrypt unregistered [11136.545470] Key type ._llcrypt registered [11136.549459] Key type .llcrypt registered [11136.710941] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11138.212071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11138.252456] alg: No test for adler32 (adler32-zlib) [11139.303783] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [11139.517577] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11141.216367] Key type lgssc registered [11142.487765] Lustre: Echo OBD driver; http://www.lustre.org/ [11150.808547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11150.835406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11152.272123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11156.732548] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11162.599047] LustreError: 313696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11162.622359] LustreError: 313696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11162.795075] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11167.716267] LustreError: 313695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.749929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.219298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11175.208181] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11186.256909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11186.711428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11194.142762] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11194.925592] LustreError: 315245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.971119] LustreError: 315248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.005321] LustreError: 315248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11197.007092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11200.376696] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11202.028871] LustreError: 315245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.077244] LustreError: 315245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11208.351372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11208.669962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11213.843424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11215.452848] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11221.892240] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11229.800039] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11230.010065] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11336.678132] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11336.683101] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11336.710571] LustreError: Skipped 1 previous similar message [11336.738601] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11336.741622] Lustre: Skipped 1 previous similar message [11341.435275] Lustre: server umount lustre-OST0001 complete [11341.798187] LustreError: 315247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.814050] LustreError: 315247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11348.449445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11348.464137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11348.478245] Lustre: Skipped 1 previous similar message [11348.490227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11350.504833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11350.516435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11350.537787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11354.358466] Lustre: server umount lustre-OST0000 complete [11359.715570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11359.730227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.754266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11359.761593] Lustre: Skipped 2 previous similar messages [11364.303695] Lustre: server umount lustre-MDT0000 complete [11367.394097] LustreError: 313691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.419073] LustreError: 313691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11368.147064] LustreError: 313675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238457 with bad export cookie 4206453831059610470 [11368.149538] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11368.158431] LustreError: 313675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11368.594563] Lustre: server umount lustre-MDT0001 complete [11379.680963] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11382.947852] Key type lgssc unregistered [11383.288628] LNet: 318248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11383.295606] LNetError: 318248:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11383.310068] LNet: Removed LNI 192.168.204.131@tcp [11384.254304] Key type .llcrypt unregistered [11384.256320] Key type ._llcrypt unregistered [11436.402776] Key type ._llcrypt registered [11436.407866] Key type .llcrypt registered [11436.515584] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11437.603749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11437.772619] alg: No test for adler32 (adler32-zlib) [11438.813973] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [11439.059865] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11440.720189] Key type lgssc registered [11441.679871] Lustre: Echo OBD driver; http://www.lustre.org/ [11451.473878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11451.491578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11452.992469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11456.777110] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11463.141531] LustreError: 320524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11463.172542] LustreError: 320524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11466.094652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.580163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11471.525227] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11474.992728] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11482.104348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.362444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.849398] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11491.626867] LustreError: 321980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.957021] LustreError: 321978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11498.000029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11498.656919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11498.925101] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11504.120218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11505.900680] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11515.664812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11523.356070] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11538.402280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11538.409698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.424376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.939497] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.945785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.964854] Lustre: Skipped 2 previous similar messages [11539.970169] Lustre: Skipped 2 previous similar messages [11543.057777] Lustre: server umount lustre-MDT0000 complete [11545.072993] LustreError: 320524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.110779] LustreError: 320524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11547.076993] LustreError: 320505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238636 with bad export cookie 6995158220102662148 [11547.078596] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11547.088669] LustreError: 320505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11547.444591] Lustre: server umount lustre-MDT0001 complete [11562.391771] Lustre: server umount lustre-OST0000 complete [11575.891208] Lustre: server umount lustre-OST0001 complete [11594.024655] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11604.925924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.569642] LustreError: 325706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11605.674691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11610.668462] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11615.211814] LustreError: 325707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.232199] LustreError: 325707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11620.218116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11620.582638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11625.073755] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11627.870398] Lustre: 326813:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11634.477213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11641.882329] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11645.109612] LustreError: 327168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.150181] LustreError: 327168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11650.304193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11651.623123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11652.044253] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11652.051839] Lustre: Skipped 1 previous similar message [11657.220840] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11658.747500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11666.448774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11671.189967] Lustre: 328651:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11677.153476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11677.163316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11677.178213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.460380] Lustre: server umount lustre-MDT0000 complete [11682.793152] LustreError: 325701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11682.813790] LustreError: 325701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11686.421677] LustreError: 327977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238775 with bad export cookie 6995158220102663849 [11686.430787] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.437572] LustreError: 327977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11686.925413] Lustre: server umount lustre-MDT0001 complete [11701.627515] Lustre: server umount lustre-OST0000 complete [11704.288636] Lustre: 319788:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781238777/real 1781238777] req@ffff8e000b7ce940 x1867763970852352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781238793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11704.323444] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.340988] Lustre: Skipped 3 previous similar messages [11706.319657] Lustre: server umount lustre-OST0001 complete [11710.032804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11712.116868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11714.281987] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 00:33:21 (1781238801) [11738.529445] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11748.471629] loop0: detected capacity change from 0 to 1099511627776 [11749.105759] SGI XFS with ACLs, security attributes, quota, no debug enabled [11749.116424] *** VALIDATE xfs *** [11749.137913] XFS (loop0): Mounting V5 Filesystem [11749.189825] XFS (loop0): Ending clean mount [11750.480333] loop1: detected capacity change from 0 to 18691697672192 [11878.370562] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [11888.543342] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11930.402968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.940774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11930.970383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11931.101416] Lustre: lustre-MDT0000: new disk, initializing [11931.257423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11931.293686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11935.556099] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11947.687605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11947.780548] Lustre: 333741:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11947.822414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11947.826348] Lustre: Skipped 1 previous similar message [11947.888432] Lustre: lustre-MDT0001: new disk, initializing [11947.988213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11947.999928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11953.490441] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11962.397889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11967.010437] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11974.466433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11974.770640] Lustre: lustre-OST0000: new disk, initializing [11974.778255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11974.783938] Lustre: 335485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11976.330952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11976.343468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11976.486034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11982.018690] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11989.635967] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11994.599978] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11994.845624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11999.487710] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11999.777511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12002.277950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12002.278297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.279677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.279687] Lustre: Skipped 3 previous similar messages [12002.299469] Lustre: Skipped 1 previous similar message [12007.399273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12007.409481] Lustre: Skipped 1 previous similar message [12016.096245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12016.290881] Lustre: server umount lustre-OST0000 complete [12022.756278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.761431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.778459] Lustre: Skipped 1 previous similar message [12022.792329] Lustre: Skipped 4 previous similar messages [12035.042219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12035.293705] Lustre: server umount lustre-MDT0000 complete [12038.121714] LustreError: 333751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12038.121714] LustreError: 333746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12039.670357] LustreError: 334527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239129 with bad export cookie 6995158220102665403 [12039.671608] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12039.687654] LustreError: 334527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12040.191541] Lustre: server umount lustre-MDT0001 complete [12049.215147] XFS (loop0): Unmounting Filesystem [12054.471253] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 00:39:01 (1781239141) [12062.425707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.298751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12063.315230] Lustre: Skipped 2 previous similar messages [12069.874407] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12082.597929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12089.399667] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12097.326101] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12101.366526] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12108.652709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12115.795625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12121.800767] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12149.228028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.229568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12149.249271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.260519] LustreError: Skipped 2 previous similar messages [12149.266260] Lustre: Skipped 5 previous similar messages [12154.044786] Lustre: server umount lustre-OST0000 complete [12164.513375] Lustre: server umount lustre-MDT0000 complete [12168.685304] LustreError: 337893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.700197] LustreError: 337893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [12168.985298] LustreError: 337879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239258 with bad export cookie 6995158220102666285 [12168.999660] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12169.006874] LustreError: 337879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12169.455660] Lustre: server umount lustre-MDT0001 complete [12181.295486] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12185.030473] Key type lgssc unregistered [12185.359623] LNet: 341195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12185.368054] LNetError: 341195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12185.394254] LNet: Removed LNI 192.168.204.131@tcp [12186.458264] Key type .llcrypt unregistered [12186.461395] Key type ._llcrypt unregistered [12217.396048] Key type ._llcrypt registered [12217.400449] Key type .llcrypt registered [12217.510026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12218.911742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12218.940388] alg: No test for adler32 (adler32-zlib) [12220.032984] Lustre: Lustre: Build Version: 2.17.53_67_g28bb009 [12220.373620] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12222.112206] Key type lgssc registered [12223.343902] Lustre: Echo OBD driver; http://www.lustre.org/ [12232.131616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12232.176681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12233.928362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12239.063501] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12244.453612] LustreError: 342747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.473224] LustreError: 342747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12248.942586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12249.280677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12253.634990] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12259.363507] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12262.781368] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12270.574152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12270.933549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12278.960198] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12285.557710] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12294.634389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12294.643421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12294.652671] Lustre: Skipped 1 previous similar message [12298.819321] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [12298.829360] Lustre: Skipped 1 previous similar message [12300.663202] Lustre: server umount lustre-MDT0001 complete [12303.916828] LustreError: 344853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12309.019091] LustreError: 342742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12309.033667] LustreError: 342742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12314.137982] LustreError: 344853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12314.158849] LustreError: 344853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12314.192763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.356304] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12314.365121] Lustre: Found index 1 for lustre-MDT0001, updating log [12314.444780] Lustre: 342737:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12314.949527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12320.238323] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12321.297231] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12322.861652] LustreError: 345914:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12322.877846] LustreError: 345914:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12322.894599] LustreError: 342739:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12322.972245] LustreError: 345919:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -17 [12322.987059] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12322.987059] [12631.633365] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [12631.642464] Lustre: Skipped 2 previous similar messages [12632.552543] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12632.563849] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12633.904665] Lustre: server umount lustre-MDT0001 complete [12636.700128] LustreError: 344032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.331069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12645.447337] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12645.459351] Lustre: Found index 1 for lustre-MDT0001, updating log [12645.533742] Lustre: 342737:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12645.995654] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12650.923888] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12651.503250] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12655.618252] LustreError: 354101:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12655.635177] LustreError: 354101:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12655.664367] LustreError: 342739:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12655.667651] LustreError: 342739:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12655.769406] LustreError: 354104:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12655.781216] LustreError: 354104:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12655.789592] LustreError: 354104:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -17 [12655.801199] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.131@tcp [12655.801199] [12958.701833] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12958.713402] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12958.727873] Lustre: Skipped 1 previous similar message [12958.744733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12958.751539] Lustre: Skipped 3 previous similar messages [12963.809890] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12963.825373] Lustre: Skipped 2 previous similar messages [12964.264921] Lustre: server umount lustre-MDT0001 complete [12964.383929] LustreError: 344853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12964.426211] LustreError: 344853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12976.510561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12976.662669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12976.672498] Lustre: Found index 1 for lustre-MDT0001, updating log [12976.735149] Lustre: 342736:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12977.104827] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12981.923754] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12982.274070] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12984.304672] LustreError: 361916:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12984.316808] LustreError: 361916:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12984.331045] LustreError: 342739:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12984.344032] LustreError: 342739:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12984.494394] LustreError: 361925:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12984.523728] LustreError: 361925:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -114 [12984.535212] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12984.535212] [13304.807776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13304.835318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13304.873124] Lustre: Skipped 1 previous similar message [13304.889023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13310.452808] Lustre: server umount lustre-MDT0000 complete [13312.994708] LustreError: 342743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13313.035234] LustreError: 342743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [13314.374156] LustreError: 345598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240403 with bad export cookie 10014356514881050607 [13314.380102] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13314.386213] LustreError: 345598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13318.125661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13318.156157] Lustre: Skipped 2 previous similar messages [13320.948925] Lustre: server umount lustre-MDT0001 complete [13325.550107] Lustre: server umount lustre-OST0000 complete [13335.763946] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [13347.727190] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13389.502831] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 01:01:16 (1781240476) [13396.538570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13396.890858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13396.926437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13397.047704] Lustre: lustre-MDT0000: new disk, initializing [13397.251071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13397.291386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13402.315992] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13413.359845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13413.521963] Lustre: 372266:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13413.594467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13413.601029] Lustre: Skipped 1 previous similar message [13413.692177] Lustre: lustre-MDT0001: new disk, initializing [13413.783066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13413.838199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13413.851217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13418.128646] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13428.623647] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13433.917264] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13442.949865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13443.431227] Lustre: lustre-OST0000: new disk, initializing [13443.440442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13443.447754] Lustre: 374008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13443.586747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13448.798195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13448.808395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13448.945753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13451.727836] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13460.697319] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13470.025914] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13470.036359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13470.056529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13470.063190] Lustre: Skipped 5 previous similar messages [13470.470902] Lustre: server umount lustre-MDT0000 complete [13474.792861] LustreError: 372277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13474.815531] LustreError: 372277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13475.204718] LustreError: 374007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240564 with bad export cookie 10014356514881052567 [13475.207500] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13475.224419] LustreError: 374007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13475.253205] LustreError: 373055:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13475.667943] Lustre: server umount lustre-MDT0001 complete [13489.919617] Lustre: server umount lustre-OST0000 complete [13525.552352] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [13539.607164] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13585.626430] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13281 sec ========= 01:04:32 (1781240672) [13588.237375] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:04:34 (1781240674) === [13594.290554] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:04:40 (1781240680) === [13642.638335] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13646.166978] Key type lgssc unregistered [13646.488424] LNet: 380218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13646.498364] LNetError: 380218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13646.533109] LNet: Removed LNI 192.168.204.131@tcp [13647.516363] Key type .llcrypt unregistered [13647.523256] Key type ._llcrypt unregistered