[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 593816283 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002373] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007016] pid_max: default: 32768 minimum: 301 [ 0.008169] LSM: Security Framework initializing [ 0.009070] Yama: becoming mindful. [ 0.010043] SELinux: Initializing. [ 0.011087] *** VALIDATE selinux *** [ 0.017954] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021662] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022245] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024151] *** VALIDATE tmpfs *** [ 0.025466] *** VALIDATE proc *** [ 0.027241] *** VALIDATE cgroup *** [ 0.028014] *** VALIDATE cgroup2 *** [ 0.029300] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031035] Spectre V2 : User space: Vulnerable [ 0.032016] Speculative Store Bypass: Vulnerable [ 0.035143] debug: unmapping init [mem 0xffffffffb6059000-0xffffffffb6060fff] [ 0.037761] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038941] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040031] ... version: 2 [ 0.041014] ... bit width: 48 [ 0.042018] ... generic registers: 4 [ 0.043017] ... value mask: 0000ffffffffffff [ 0.044020] ... max period: 00007fffffffffff [ 0.045011] ... fixed-purpose events: 3 [ 0.046010] ... event mask: 000000070000000f [ 0.047325] rcu: Hierarchical SRCU implementation. [ 0.049568] smp: Bringing up secondary CPUs ... [ 0.050802] x86: Booting SMP configuration: [ 0.051028] .... node #0, CPUs: #1 #2 #3 [ 0.064593] smp: Brought up 1 node, 4 CPUs [ 0.066034] smpboot: Max logical packages: 1 [ 0.067013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131038] node 0 deferred pages initialised in 59ms [ 0.136828] devtmpfs: initialized [ 0.137263] x86/mm: Memory block size: 128MB [ 0.139594] gcov: version magic: 0x41383552 [ 0.141300] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143263] pinctrl core: initialized pinctrl subsystem [ 0.144167] [ 0.144905] ************************************************************* [ 0.145012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146016] ** ** [ 0.147010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148012] ** ** [ 0.149015] ** This means that this kernel is built to expose internal ** [ 0.150010] ** IOMMU data structures, which may compromise security on ** [ 0.151013] ** your system. ** [ 0.152013] ** ** [ 0.153012] ** If you see this message and you are not debugging the ** [ 0.154012] ** kernel, report this immediately to your vendor! ** [ 0.155009] ** ** [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157012] ************************************************************* [ 0.158949] NET: Registered protocol family 16 [ 0.159459] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.160055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162701] cpuidle: using governor menu [ 0.165318] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.166533] PCI: Using configuration type 1 for base access [ 0.167212] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.179090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.180013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184073] cryptd: max_cpu_qlen set to 1000 [ 0.188325] ACPI: Added _OSI(Module Device) [ 0.190013] ACPI: Added _OSI(Processor Device) [ 0.191009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197676] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204491] ACPI: Interpreter enabled [ 0.206162] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208009] ACPI: Using IOAPIC for interrupt routing [ 0.210120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213357] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223963] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236347] acpiphp: Slot [2] registered [ 0.238085] acpiphp: Slot [3] registered [ 0.239062] acpiphp: Slot [4] registered [ 0.240186] acpiphp: Slot [5] registered [ 0.242090] acpiphp: Slot [6] registered [ 0.243081] acpiphp: Slot [7] registered [ 0.245082] acpiphp: Slot [8] registered [ 0.246074] acpiphp: Slot [9] registered [ 0.248081] acpiphp: Slot [10] registered [ 0.249111] acpiphp: Slot [11] registered [ 0.251075] acpiphp: Slot [12] registered [ 0.252250] acpiphp: Slot [13] registered [ 0.254088] acpiphp: Slot [14] registered [ 0.255216] acpiphp: Slot [15] registered [ 0.257081] acpiphp: Slot [16] registered [ 0.259085] acpiphp: Slot [17] registered [ 0.261108] acpiphp: Slot [18] registered [ 0.262075] acpiphp: Slot [19] registered [ 0.264103] acpiphp: Slot [20] registered [ 0.265087] acpiphp: Slot [21] registered [ 0.267079] acpiphp: Slot [22] registered [ 0.268075] acpiphp: Slot [23] registered [ 0.270102] acpiphp: Slot [24] registered [ 0.271147] acpiphp: Slot [25] registered [ 0.273077] acpiphp: Slot [26] registered [ 0.274119] acpiphp: Slot [27] registered [ 0.276087] acpiphp: Slot [28] registered [ 0.278092] acpiphp: Slot [29] registered [ 0.279196] acpiphp: Slot [30] registered [ 0.281116] acpiphp: Slot [31] registered [ 0.283045] PCI host bridge to bus 0000:00 [ 0.284012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286059] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295017] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.297017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299710] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302381] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307506] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320634] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.323014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330518] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.335034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338698] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342996] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.353012] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.357013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.369014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.375026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.389014] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.404329] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.412014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441013] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.449727] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.454012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.459014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.473015] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.483451] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.498015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.505015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.522014] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.533250] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.542014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.552017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.571013] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.579469] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.592017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.601016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.616015] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.628000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.631334] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.633288] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.635290] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.641278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.646191] iommu: Default domain type: Passthrough [ 0.647000] SCSI subsystem initialized [ 0.647110] ACPI: bus type USB registered [ 0.648000] usbcore: registered new interface driver usbfs [ 0.650047] usbcore: registered new interface driver hub [ 0.652059] usbcore: registered new device driver usb [ 0.653153] pps_core: LinuxPPS API ver. 1 registered [ 0.655008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.657046] PTP clock support registered [ 0.659108] EDAC MC: Ver: 3.0.0 [ 0.661000] PCI: Using ACPI for IRQ routing [ 0.663419] NetLabel: Initializing [ 0.665013] NetLabel: domain hash size = 128 [ 0.668015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.672085] NetLabel: unlabeled traffic allowed by default [ 0.673042] vgaarb: loaded [ 0.675239] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.676000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688000] clocksource: Switched to clocksource kvm-clock [ 0.796134] VFS: Disk quotas dquot_6.6.0 [ 0.797506] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799698] *** VALIDATE ramfs *** [ 0.800775] *** VALIDATE hugetlbfs *** [ 0.802182] pnp: PnP ACPI init [ 0.804269] pnp: PnP ACPI: found 6 devices [ 0.825078] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.827919] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.829841] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.834021] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.836136] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.838182] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.840549] NET: Registered protocol family 2 [ 0.842683] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.846962] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.850222] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.854800] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.857769] TCP: Hash tables configured (established 65536 bind 65536) [ 0.860282] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.863773] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866262] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869202] NET: Registered protocol family 1 [ 0.876554] RPC: Registered named UNIX socket transport module. [ 0.878516] RPC: Registered udp transport module. [ 0.880792] RPC: Registered tcp transport module. [ 0.882339] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.884386] NET: Registered protocol family 44 [ 0.885801] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.887736] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.889612] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.891689] PCI: CLS 0 bytes, default 64 [ 0.893750] Unpacking initramfs... [ 2.369519] debug: unmapping init [mem 0xffffa0a8fcc54000-0xffffa0a8fffbffff] [ 2.374114] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.376497] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.379500] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.889232] Initialise system trusted keyrings [ 2.890679] Key type blacklist registered [ 2.892478] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.901668] zbud: loaded [ 2.905432] *** VALIDATE nfs *** [ 2.906497] *** VALIDATE nfs4 *** [ 2.907962] pstore: using deflate compression [ 2.911436] Platform Keyring initialized [ 3.056842] NET: Registered protocol family 38 [ 3.058881] Key type asymmetric registered [ 3.060743] Asymmetric key parser 'x509' registered [ 3.062905] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.067088] io scheduler mq-deadline registered [ 3.068963] io scheduler kyber registered [ 3.070910] io scheduler bfq registered [ 3.072881] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.076714] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.082431] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.085474] ACPI: Power Button [PWRF] [ 3.191763] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.299207] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.530499] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.658682] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.887977] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.921158] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.960141] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.965855] Non-volatile memory driver v1.3 [ 3.970859] Linux agpgart interface v0.103 [ 4.016072] virtio_blk virtio1: [vda] 133256 512-byte logical blocks (68.2 MB/65.1 MiB) [ 4.019573] vda: detected capacity change from 0 to 68227072 [ 4.038513] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.043433] vdb: detected capacity change from 0 to 1073741824 [ 4.060810] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.063615] vdc: detected capacity change from 0 to 2621440000 [ 4.090580] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.092578] vdd: detected capacity change from 0 to 2621440000 [ 4.108905] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.113641] vde: detected capacity change from 0 to 4294967296 [ 4.135837] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.144208] vdf: detected capacity change from 0 to 4294967296 [ 4.157394] libphy: Fixed MDIO Bus: probed [ 4.164542] usbcore: registered new interface driver usbserial_generic [ 4.166597] usbserial: USB Serial support registered for generic [ 4.168602] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.176447] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.177854] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.179882] mousedev: PS/2 mouse device common for all mice [ 4.182663] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.184241] rtc_cmos 00:05: RTC can wake from S4 [ 4.191841] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.197975] rtc_cmos 00:05: registered as rtc0 [ 4.199903] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.205570] intel_pstate: CPU model not supported [ 4.209636] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.214642] hid: raw HID events driver (C) Jiri Kosina [ 4.218427] usbcore: registered new interface driver usbhid [ 4.221553] usbhid: USB HID core driver [ 4.223722] drop_monitor: Initializing network drop monitor service [ 4.228188] Initializing XFRM netlink socket [ 4.230181] NET: Registered protocol family 10 [ 4.232975] Segment Routing with IPv6 [ 4.234467] NET: Registered protocol family 17 [ 4.238162] mpls_gso: MPLS GSO support [ 4.245367] RAS: Correctable Errors collector initialized. [ 4.247273] AVX version of gcm_enc/dec engaged. [ 4.249579] AES CTR mode by8 optimization enabled [ 4.372740] sched_clock: Marking stable (4372724462, 0)->(5479187939, -1106463477) [ 4.375873] registered taskstats version 1 [ 4.378503] Loading compiled-in X.509 certificates [ 4.380102] zswap: loaded using pool lzo/zbud [ 4.414507] Key type big_key registered [ 4.430417] Key type encrypted registered [ 4.431644] ima: No TPM chip found, activating TPM-bypass! [ 4.433231] ima: Allocated hash algorithm: sha1 [ 4.435815] ima: No architecture policies found [ 4.437495] evm: Initialising EVM extended attributes: [ 4.439069] evm: security.selinux [ 4.446249] evm: security.ima [ 4.446991] evm: security.capability [ 4.447930] evm: HMAC attrs: 0x1 [ 4.456425] rtc_cmos 00:05: setting system clock to 2025-07-14 16:15:04 UTC (1752509704) [ 4.463569] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 4.475078] debug: unmapping init [mem 0xffffffffb5d82000-0xffffffffb6058fff] [ 4.487056] Write protecting the kernel read-only data: 28672k [ 4.490173] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 4.492599] debug: unmapping init [mem 0xffffffffb4d14000-0xffffffffb4dfffff] [ 4.549798] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.557456] systemd[1]: Detected virtualization kvm. [ 4.559321] systemd[1]: Detected architecture x86-64. [ 4.561549] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.598169] systemd[1]: No hostname configured. [ 4.599764] systemd[1]: Set hostname to . [ 4.601674] random: systemd: uninitialized urandom read (16 bytes read) [ 4.603772] systemd[1]: Initializing machine ID from random generator. [ 4.697659] random: ln: uninitialized urandom read (6 bytes read) [ 4.839581] random: systemd: uninitialized urandom read (16 bytes read) [ 4.847367] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.862556] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.877613] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.985509] device-mapper: uevent: version 1.0.3 [ 5.991575] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 7.061131] random: fast init done [ 7.137961] virtio_net virtio0 ens2: renamed from eth0 [ 7.685797] scsi host0: ata_piix [ 7.695431] scsi host1: ata_piix [ 7.702878] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.713484] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.965818] random: crng init done [ 11.967076] random: 7 urandom warning(s) missed due to ratelimiting [ 13.157990] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.723685] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.535898] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.058551] SELinux: Disabled at runtime. [ 17.132096] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.147861] systemd[1]: Detected virtualization kvm. [ 17.149891] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.078326] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.086158] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.094488] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.113442] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.123639] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.145221] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.168751] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 18.299464] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.976290] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.561106] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.611616] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.029870] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.100430] EDAC sbridge: Ver: 1.1.2 [ 22.753740] Key type dns_resolver registered [ 23.103607] NFS: Registering the id_resolver key type [ 23.105521] Key type id_resolver registered [ 23.108467] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg334-server login: [ 58.036667] libcfs: loading out-of-tree module taints kernel. [ 58.047062] Key type ._llcrypt registered [ 58.048211] Key type .llcrypt registered [ 58.089068] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 66.393805] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 67.011320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.018117] alg: No test for adler32 (adler32-zlib) [ 68.040442] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 68.379298] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 68.381321] LNet: Accept secure, port 988 [ 70.015171] Key type lgssc registered [ 70.708968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.242991] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 79.413904] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.625136] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.453089] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.514533] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.018489] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 101.604534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.628527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 101.641690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.753330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 102.770563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 102.815342] Lustre: lustre-MDT0000: new disk, initializing [ 102.852758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.863777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 104.461887] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.708415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.756827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 110.797463] Lustre: 6452:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 110.814524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 110.817576] Lustre: Skipped 1 previous similar message [ 110.863994] Lustre: lustre-MDT0001: new disk, initializing [ 110.890957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 110.906364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 110.911215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 112.433436] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.030928] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 119.030458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.070447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 119.179773] Lustre: lustre-OST0000: new disk, initializing [ 119.182722] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 119.209168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.447729] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.459129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 126.462482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 126.511356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 127.841711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.870180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 127.917258] Lustre: lustre-OST0001: new disk, initializing [ 127.920280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 127.939839] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.120294] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.648175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 134.652288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 134.679427] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 136.373499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 142.225865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 149.603862] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing check_logdir /tmp/testlogs/ [ 151.589655] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing yml_node [ 153.882175] Lustre: DEBUG MARKER: Client: 2.16.57.2 [ 155.111583] Lustre: DEBUG MARKER: MDS: 2.16.57.2 [ 156.278452] Lustre: DEBUG MARKER: OSS: 2.16.57.2 [ 157.014347] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 14 12:17:35 EDT 2025 [ 165.178572] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 165.907856] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 180.704053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 180.708035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 180.713917] Lustre: Skipped 1 previous similar message [ 180.718397] Lustre: Skipped 3 previous similar messages [ 184.393861] LustreError: 12119:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 184.456880] Lustre: server umount lustre-MDT0000 complete [ 186.352657] LustreError: 6445:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752509886 with bad export cookie 11714746209065231253 [ 186.353941] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 186.359032] LustreError: 6445:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 190.943490] LustreError: 6460:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 190.943944] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.944359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.953135] LustreError: 6460:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 190.966118] Lustre: Skipped 3 previous similar messages [ 191.968484] LustreError: 6459:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 191.969891] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 191.980397] LustreError: 6459:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 191.985118] Lustre: Skipped 2 previous similar messages [ 192.521291] LustreError: 12321:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 192.523203] LustreError: 12321:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 192.632865] Lustre: server umount lustre-MDT0001 complete [ 194.727106] LustreError: 12523:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 194.735224] LustreError: 12523:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 194.768149] Lustre: server umount lustre-OST0000 complete [ 196.743275] LustreError: 12723:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 196.747639] LustreError: 12723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 196.826367] Lustre: server umount lustre-OST0001 complete [ 199.869272] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 204.542373] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 209.608710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.845576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.863126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.897316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.100110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.146620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.257314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 222.274415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 222.317398] Lustre: lustre-MDT0000: new disk, initializing [ 222.381939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.392153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.193182] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 229.899909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.940045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 229.988869] Lustre: 14887:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.135170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.911133] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 235.839688] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 237.494348] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.138232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 241.175230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 241.313577] Lustre: lustre-OST0000: new disk, initializing [ 241.316125] Lustre: Skipped 1 previous similar message [ 241.320252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.323577] Lustre: Skipped 2 previous similar messages [ 241.375551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.378505] Lustre: Skipped 1 previous similar message [ 242.572984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.577399] Lustre: Skipped 1 previous similar message [ 242.580288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.633072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.310289] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 248.383345] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 250.293878] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 250.394880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 252.039725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 252.146168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 258.018215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 258.019517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.021405] LustreError: Skipped 1 previous similar message [ 258.028671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 258.032700] Lustre: Skipped 1 previous similar message [ 259.125165] LustreError: 17793:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 259.129630] LustreError: 17793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 259.209890] Lustre: server umount lustre-OST0000 complete [ 261.090032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 261.094052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.099617] Lustre: Skipped 1 previous similar message [ 263.136200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 263.138876] Lustre: Skipped 2 previous similar messages [ 267.141264] Lustre: server umount lustre-MDT0000 complete [ 268.256711] LustreError: 14898:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 268.264680] LustreError: 14898:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 268.892645] LustreError: 14879:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752509968 with bad export cookie 11714746209065232772 [ 268.897985] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.898387] LustreError: 14879:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 268.965376] LustreError: 18195:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 268.967634] LustreError: 18195:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 269.060327] Lustre: server umount lustre-MDT0001 complete [ 271.235674] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:19:29 (1752509969) [ 274.462933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.723637] LustreError: 18721:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.773607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.627094] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 280.036609] LustreError: 18721:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.917277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 282.786533] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 285.316064] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 286.967678] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 289.986618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 292.784591] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 295.144326] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 302.453636] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 306.656844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 306.660240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.665283] Lustre: Skipped 1 previous similar message [ 306.668033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.935187] LustreError: 21002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 311.937094] LustreError: 21002:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 311.967632] Lustre: server umount lustre-MDT0000 complete [ 313.419413] LustreError: 18701:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510013 with bad export cookie 11714746209065233654 [ 313.423905] LustreError: 18701:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 313.425150] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.588222] Lustre: server umount lustre-MDT0001 complete [ 325.518386] Lustre: server umount lustre-OST0000 complete [ 334.682292] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 338.680370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.884596] LustreError: 22724:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.909959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.912677] Lustre: Skipped 2 previous similar messages [ 340.479910] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 344.168973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.926538] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 347.160235] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 350.014535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 352.664187] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 356.360919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 356.393768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 356.466880] Lustre: lustre-OST0001: new disk, initializing [ 356.470146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 357.773117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 357.778059] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 357.820255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 358.953300] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 364.438687] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 366.085096] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 378.026524] LustreError: 26228:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 378.028513] LustreError: 26228:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 378.068281] Lustre: server umount lustre-MDT0000 complete [ 379.362852] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.364237] LustreError: 23967:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.375709] Lustre: Skipped 5 previous similar messages [ 379.386179] LustreError: 23967:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 379.664730] LustreError: 26229:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510079 with bad export cookie 11714746209065234501 [ 379.667394] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.670394] LustreError: 26229:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 384.483921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.489305] Lustre: Skipped 4 previous similar messages [ 385.883735] Lustre: server umount lustre-MDT0001 complete [ 387.482275] Lustre: server umount lustre-OST0000 complete [ 389.142864] Lustre: server umount lustre-OST0001 complete [ 395.524578] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 399.396512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.219326] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 404.722365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 404.924522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.927162] Lustre: Skipped 4 previous similar messages [ 406.461368] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 407.632321] Lustre: 29095:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 410.419715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 412.780036] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 414.693282] LustreError: 29453:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.700843] LustreError: 29453:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 416.266723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 418.609340] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 427.633550] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 428.982602] Lustre: 30937:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 440.292272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.293777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.297624] Lustre: Skipped 5 previous similar messages [ 440.300472] Lustre: Skipped 2 previous similar messages [ 441.888190] Lustre: server umount lustre-MDT0000 complete [ 443.282567] LustreError: 29097:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510143 with bad export cookie 11714746209065235796 [ 443.285974] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.287613] LustreError: 29097:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 443.362344] LustreError: 31341:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 443.364951] LustreError: 31341:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 443.450313] Lustre: server umount lustre-MDT0001 complete [ 455.628157] Lustre: server umount lustre-OST0000 complete [ 467.523685] Lustre: server umount lustre-OST0001 complete [ 470.919487] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:22:49 (1752510169) [ 481.570441] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 485.292720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.453981] LustreError: 33708:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.969037] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 490.280637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 492.026746] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 493.157555] Lustre: 34814:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 495.824925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 498.149794] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 501.498907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 503.959763] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 518.056160] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 519.573563] Lustre: 36677:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 526.304366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.307686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.312696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.315297] Lustre: Skipped 1 previous similar message [ 532.316526] Lustre: server umount lustre-MDT0000 complete [ 533.710844] LustreError: 33689:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510233 with bad export cookie 11714746209065236902 [ 533.715833] LustreError: 33689:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 533.717403] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.878615] Lustre: server umount lustre-MDT0001 complete [ 545.285133] Lustre: server umount lustre-OST0000 complete [ 557.058883] Lustre: server umount lustre-OST0001 complete [ 560.260331] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:24:19 (1752510259) [ 566.058458] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 569.897296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.112036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.115961] Lustre: Skipped 6 previous similar messages [ 571.656272] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 575.141095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 576.906256] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 578.089573] Lustre: 39895:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 580.767247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 583.130129] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 586.384994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 588.691901] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 602.912739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 609.607893] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 609.610282] Lustre: Skipped 1 previous similar message [ 611.296407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.297259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 611.300230] Lustre: Skipped 5 previous similar messages [ 616.833922] LustreError: 41961:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 616.836783] LustreError: 41961:0:(obd_class.h:479:obd_check_dev()) Skipped 43 previous similar messages [ 616.867985] Lustre: server umount lustre-MDT0000 complete [ 618.275345] LustreError: 41760:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510318 with bad export cookie 11714746209065238008 [ 618.279071] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.280367] LustreError: 41760:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 620.590930] LustreError: 41063:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.598317] LustreError: 41063:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 624.468922] Lustre: server umount lustre-MDT0001 complete [ 627.804284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.704096] LustreError: 3622:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0a842adce00 x1837639682813440/t0(0) o250->MGC192.168.203.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 630.494786] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 633.850931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 635.588789] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 637.728083] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 647.621665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 648.679207] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 649.023300] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 649.718919] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 657.518699] Lustre: server umount lustre-MDT0000 complete [ 659.066604] LustreError: 38772:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510359 with bad export cookie 11714746209065239149 [ 659.071825] LustreError: 38772:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.072065] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.241588] Lustre: server umount lustre-MDT0001 complete [ 670.212597] Lustre: server umount lustre-OST0000 complete [ 681.998785] Lustre: server umount lustre-OST0001 complete [ 685.727188] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:26:24 (1752510384) [ 691.899571] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 696.076264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.823934] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 701.301582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 701.408156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 701.411135] Lustre: Skipped 22 previous similar messages [ 703.054068] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 704.272485] Lustre: 47209:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 704.277305] Lustre: 47209:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 706.965398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 709.502081] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 712.905323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 715.304315] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 718.886776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 725.693058] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 732.128434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 736.746605] Lustre: server umount lustre-MDT0000 complete [ 738.244521] LustreError: 46083:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510438 with bad export cookie 11714746209065239947 [ 738.249183] LustreError: 46083:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 738.250193] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 738.419278] Lustre: server umount lustre-MDT0001 complete [ 750.210861] Lustre: server umount lustre-OST0000 complete [ 761.996914] Lustre: server umount lustre-OST0001 complete [ 765.741416] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:27:44 (1752510464) [ 766.745770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 767.465892] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:27:46 (1752510466) [ 771.421155] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 774.622500] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 778.420249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 780.721853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 783.462852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 783.491962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.598773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 783.639404] Lustre: lustre-MDT0000: new disk, initializing [ 783.686287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 785.150293] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 788.587220] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 791.211056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 791.240534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 793.291254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 793.306490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 793.617917] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 797.027165] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.876338] Lustre: DEBUG MARKER: create test files [ 814.049586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.049669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.057446] Lustre: Skipped 9 previous similar messages [ 816.310710] Lustre: server umount lustre-OST0000 complete [ 817.840668] Lustre: server umount lustre-MDT0000 complete [ 823.186457] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 824.425243] Key type lgssc unregistered [ 824.578751] LNet: 54571:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.637605] LNet: Removed LNI 192.168.203.134@tcp [ 826.029550] Key type .llcrypt unregistered [ 826.031201] Key type ._llcrypt unregistered [ 837.918843] Key type ._llcrypt registered [ 837.920268] Key type .llcrypt registered [ 837.967938] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 838.297470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.336035] alg: No test for adler32 (adler32-zlib) [ 839.223660] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 839.328703] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 839.331606] LNet: Accept secure, port 988 [ 840.927223] Key type lgssc registered [ 841.364873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 844.412113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 844.422184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.617104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.125798] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 849.224245] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 851.831674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 851.967721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.167519] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 855.077740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 855.108972] Lustre: 57136:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 855.112990] Lustre: 57136:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 855.116335] Lustre: 57136:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 855.119377] Lustre: 57136:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 855.123117] Lustre: 57136:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 855.126820] Lustre: 57136:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 856.171317] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 859.327738] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 861.121205] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 865.249399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.249506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.256906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 869.231081] LustreError: 57873:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 869.294496] Lustre: server umount lustre-OST0000 complete [ 870.709036] LustreError: 58074:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 870.711597] LustreError: 58074:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 870.798197] Lustre: server umount lustre-MDT0000 complete [ 875.689952] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 876.836376] Key type lgssc unregistered [ 876.987616] LNet: 58601:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.053665] LNet: Removed LNI 192.168.203.134@tcp [ 878.397876] Key type .llcrypt unregistered [ 878.399350] Key type ._llcrypt unregistered [ 890.407927] Key type ._llcrypt registered [ 890.409731] Key type .llcrypt registered [ 890.449804] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 890.815377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 890.855317] alg: No test for adler32 (adler32-zlib) [ 891.721406] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 891.812327] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 891.815194] LNet: Accept secure, port 988 [ 893.399124] Key type lgssc registered [ 893.797420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.665834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 896.674261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.836820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.264319] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 901.315292] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.878395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.011824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.125213] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 907.108110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 907.131139] Lustre: 61266:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 907.134915] Lustre: 61266:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 907.138416] Lustre: 61266:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 907.141936] Lustre: 61266:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 907.145251] Lustre: 61266:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 907.148652] Lustre: 61266:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 908.058885] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 911.122906] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 915.502077] LustreError: 61951:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 915.558054] Lustre: server umount lustre-OST0000 complete [ 926.179362] LustreError: 62152:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 926.181415] LustreError: 62152:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 926.272686] Lustre: server umount lustre-MDT0000 complete [ 931.058235] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 932.205469] Key type lgssc unregistered [ 932.352432] LNet: 62682:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.413518] LNet: Removed LNI 192.168.203.134@tcp [ 933.755509] Key type .llcrypt unregistered [ 933.756981] Key type ._llcrypt unregistered [ 939.958060] Key type ._llcrypt registered [ 939.959318] Key type .llcrypt registered [ 940.003852] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 946.370896] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 946.786666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.792525] alg: No test for adler32 (adler32-zlib) [ 947.643074] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 947.725579] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 947.728259] LNet: Accept secure, port 988 [ 949.311145] Key type lgssc registered [ 949.696600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.911680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 955.361102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 957.689820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 960.057820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 962.614679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 962.633310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 962.642058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.738197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 963.752984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 963.793374] Lustre: lustre-MDT0000: new disk, initializing [ 963.826422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.834983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 965.261908] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 969.548603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 969.571839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 969.608808] Lustre: 65764:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 969.638678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 969.641282] Lustre: Skipped 1 previous similar message [ 969.680574] Lustre: lustre-MDT0001: new disk, initializing [ 969.716306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 969.732586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 969.736737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 971.176457] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 974.461390] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 975.829843] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 978.276968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 978.308196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 978.401657] Lustre: lustre-OST0000: new disk, initializing [ 978.403850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 978.431718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.702233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 978.706344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 978.756238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 980.471666] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 983.705177] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 985.106849] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 985.193194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 986.580289] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 986.666314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 989.154325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.154657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.155307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.157225] LustreError: Skipped 1 previous similar message [ 989.160852] Lustre: Skipped 1 previous similar message [ 993.391158] LustreError: 68671:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 993.452794] Lustre: server umount lustre-OST0000 complete [ 995.296062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 995.299167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.304919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.307188] Lustre: Skipped 1 previous similar message [ 999.392397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.393075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.512188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.514937] Lustre: Skipped 2 previous similar messages [ 1009.119172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1009.159170] LustreError: 68872:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1009.161908] LustreError: 68872:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1009.193380] Lustre: server umount lustre-MDT0000 complete [ 1010.670043] LustreError: 65757:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510710 with bad export cookie 6318653204906669605 [ 1010.673957] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.676223] LustreError: 65757:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1010.746448] LustreError: 69076:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 1010.749334] LustreError: 69076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1010.839184] Lustre: server umount lustre-MDT0001 complete [ 1014.067260] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 12:31:52 (1752510712) [ 1017.876251] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1022.766981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1031.973313] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 1035.509884] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1039.252562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1041.799046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1044.109709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1046.579634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1049.273254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1049.297792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.389117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1049.402275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1049.444693] Lustre: lustre-MDT0000: new disk, initializing [ 1049.474708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.485360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1050.956178] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1055.682979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1055.719382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1055.761740] Lustre: 73281:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1055.787198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1055.789528] Lustre: Skipped 1 previous similar message [ 1055.827656] Lustre: lustre-MDT0001: new disk, initializing [ 1055.866367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.883613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1055.889213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1057.393761] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1061.028639] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1062.464609] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1065.073926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1065.104599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1065.217740] Lustre: lustre-OST0000: new disk, initializing [ 1065.220696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1065.256526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.396436] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1070.854734] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1072.129307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1072.133172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1072.173959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1072.420788] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1072.514234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1074.061314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1074.164980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1077.219952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.221175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.223473] LustreError: Skipped 1 previous similar message [ 1077.229576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.233158] Lustre: Skipped 1 previous similar message [ 1081.006127] LustreError: 76187:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1081.009392] LustreError: 76187:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1081.079114] Lustre: server umount lustre-OST0000 complete [ 1086.943840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1086.947563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.952310] Lustre: Skipped 1 previous similar message [ 1086.954616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1088.638015] LustreError: 76389:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1088.640506] LustreError: 76389:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1088.666160] Lustre: server umount lustre-MDT0000 complete [ 1090.089611] LustreError: 75014:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510790 with bad export cookie 6318653204906670417 [ 1090.092688] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.094161] LustreError: 75014:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1090.244594] Lustre: server umount lustre-MDT0001 complete [ 1093.572754] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 12:33:12 (1752510792) [ 1096.189767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.352589] LustreError: 77259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.360112] LustreError: 77259:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1096.378808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.760531] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1101.124773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1102.848089] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1105.054406] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1106.462254] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1109.133269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1111.552618] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1113.676740] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1115.160100] Lustre: 79577:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 9c4f1ec7-f8f0-47ad-9f75-d61aec1c340b at adminstrative request [ 1116.644957] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1116.650442] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1126.885202] Lustre: MGS: Client 9c4f1ec7-f8f0-47ad-9f75-d61aec1c340b (at 0@lo) reconnecting [ 1126.888646] Lustre: Evicted from MGS (at 192.168.203.134@tcp) after server handle changed from 0x57b05d9cb9587cc3 to 0x57b05d9cb9587f4e [ 1126.894680] Lustre: MGC192.168.203.134@tcp: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 1147.868222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1147.899230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1147.961683] Lustre: lustre-OST0001: new disk, initializing [ 1147.964576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1147.996371] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1147.999041] Lustre: Skipped 2 previous similar messages [ 1150.102907] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1153.530077] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1154.566033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1154.570997] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1154.610641] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1159.648669] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.651029] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.653501] LustreError: Skipped 1 previous similar message [ 1159.658342] Lustre: Skipped 1 previous similar message [ 1159.661756] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1159.665344] Lustre: Skipped 3 previous similar messages [ 1160.748143] LustreError: 80710:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1160.750326] LustreError: 80710:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1160.773895] Lustre: server umount lustre-OST0001 complete [ 1164.771269] LustreError: 78843:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.777730] LustreError: 78843:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1168.434196] Lustre: server umount lustre-OST0000 complete [ 1184.223126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1184.258632] LustreError: 81112:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1184.260563] LustreError: 81112:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1184.290489] Lustre: server umount lustre-MDT0000 complete [ 1185.248036] LustreError: 77259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.253738] LustreError: 77259:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1185.659405] LustreError: 77238:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510885 with bad export cookie 6318653204906671950 [ 1185.665877] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.805574] Lustre: server umount lustre-MDT0001 complete [ 1188.916469] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 12:34:47 (1752510887) [ 1189.525033] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1190.169610] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 12:34:49 (1752510889) [ 1190.756385] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1191.451433] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 12:34:50 (1752510890) [ 1192.122292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1192.832936] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 12:34:51 (1752510891) [ 1193.433141] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1195.273676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1195.642129] loop: module loaded [ 1195.645870] loop0: detected capacity change from 0 to 4294967296 [ 1195.658920] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1196.142979] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1197.193214] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1198.650658] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1201.848193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1203.322281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.909909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1206.520756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1209.287995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1209.316047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1209.330837] LustreError: 83877:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1209.334305] LustreError: 83877:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1209.337468] LustreError: 83877:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1209.341214] LustreError: 83877:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1216.947508] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 1220.853673] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1224.881824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1227.593556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1229.968249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1232.390605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1234.984414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1235.021170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.118842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1235.136041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1235.175253] Lustre: lustre-MDT0000: new disk, initializing [ 1235.210793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.218037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1236.777236] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1241.415166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1241.446689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1241.485940] Lustre: 86699:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1241.605973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1243.087404] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1246.533625] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1247.923996] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1250.468853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1250.491959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1251.368823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1252.747799] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1255.984308] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1257.425686] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1257.509725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1258.907904] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1258.991942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1261.538382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1261.538652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.541269] LustreError: Skipped 3 previous similar messages [ 1261.546207] Lustre: Skipped 4 previous similar messages [ 1261.547177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.551153] Lustre: Skipped 11 previous similar messages [ 1265.772188] LustreError: 89608:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1265.775076] LustreError: 89608:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1265.838055] Lustre: server umount lustre-OST0000 complete [ 1273.371544] Lustre: server umount lustre-MDT0000 complete [ 1274.684103] LustreError: 86691:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752510974 with bad export cookie 6318653204906672552 [ 1274.686527] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.689285] LustreError: 86691:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1274.840214] Lustre: server umount lustre-MDT0001 complete [ 1277.771832] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 12:36:16 (1752510976) [ 1280.089539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.244228] LustreError: 90682:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.613484] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1284.738607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1286.295460] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1288.270202] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.622137] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1292.154987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1294.288965] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1296.280985] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.760135] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1306.043067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1307.525640] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1307.623377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1310.343526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1310.375427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1310.448067] Lustre: lustre-OST0001: new disk, initializing [ 1310.449771] Lustre: Skipped 2 previous similar messages [ 1310.452502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1310.455017] Lustre: Skipped 3 previous similar messages [ 1312.774767] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1316.302508] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1317.923187] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1320.448872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1320.452659] Lustre: Skipped 2 previous similar messages [ 1320.454932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1320.457880] Lustre: Skipped 1 previous similar message [ 1320.487336] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1321.093530] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1322.521078] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1322.609470] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1332.633418] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1340.533405] Lustre: Failing over lustre-MDT0000 [ 1340.899841] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.902965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.904852] Lustre: Skipped 6 previous similar messages [ 1340.907987] Lustre: Skipped 4 previous similar messages [ 1340.956518] LustreError: 95592:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1340.959542] LustreError: 95592:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1341.002123] Lustre: server umount lustre-MDT0000 complete [ 1344.568529] LustreError: 90676:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.574744] LustreError: 90676:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1354.799995] LustreError: 90678:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.806631] LustreError: 90678:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 11 previous similar messages [ 1356.249502] LDISKFS-fs (dm-0): recovery complete [ 1356.251401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.299450] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1356.421576] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1357.821758] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1359.919885] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1359.920501] Lustre: 90677:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1361.888618] Lustre: 90681:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1361.893495] Lustre: 90681:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 1361.899962] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 1361.918086] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1362.231195] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1362.553452] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1362.556956] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1362.871145] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1362.874519] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1363.193359] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1363.197604] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 8 previous similar messages [ 1363.825524] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1363.830780] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 1364.151116] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1364.154373] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 1365.407828] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1365.412922] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 6 previous similar messages [ 1366.040230] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1366.045133] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1366.359099] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1366.362885] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1369.493570] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1369.501618] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 12 previous similar messages [ 1370.130153] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1370.134646] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1370.455112] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1370.458182] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1377.641033] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1377.644229] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 25 previous similar messages [ 1378.273637] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1378.277457] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1378.591128] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1378.594663] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1393.889429] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1393.893627] Lustre: 96340:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 51 previous similar messages [ 1394.521198] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1394.525084] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 51 previous similar messages [ 1394.839100] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1394.842143] LustreError: 96340:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 51 previous similar messages [ 1419.191124] Lustre: 96340:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1419.193789] Lustre: 96340:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1419.196703] Lustre: 96340:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c4bfdc63-95e0-4bf1-bbe9-726bfeeebb14@192.168.203.34@tcp [ 1419.200574] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1419.202926] Lustre: 96340:0:(ldlm_lib.c:1831:abort_req_replay_queue()) @@@ aborted: req@000000009804cc6d x1837640596917120/t0(8589934780) o35->c4bfdc63-95e0-4bf1-bbe9-726bfeeebb14@192.168.203.34@tcp:599/0 lens 392/0 e 0 to 0 dl 1752511129 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1419.212332] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1419.215982] Lustre: lustre-MDT0000: Denying connection for new client c4bfdc63-95e0-4bf1-bbe9-726bfeeebb14 (at 192.168.203.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1419.219086] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1419.234753] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1419.237748] Lustre: Skipped 2 previous similar messages [ 1419.239662] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1419.254635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 1419.254635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 1424.864362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1424.867212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.723950] Lustre: server umount lustre-OST0000 complete [ 1433.570435] LustreError: 92268:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.576833] LustreError: 92268:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 1446.367110] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1446.455992] Lustre: server umount lustre-OST0001 complete [ 1462.239089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1462.306441] Lustre: server umount lustre-MDT0000 complete [ 1463.545332] LustreError: 90663:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511163 with bad export cookie 6318653204906723736 [ 1463.547724] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.548884] LustreError: 90663:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1463.693205] Lustre: server umount lustre-MDT0001 complete [ 1466.649723] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 12:39:25 (1752511165) [ 1469.156611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1469.159921] Lustre: *** cfs_fail_loc=197, val=0*** [ 1486.623504] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1492.767473] LustreError: 98290:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 1498.918940] Lustre: *** cfs_fail_loc=197, val=0*** [ 1498.919997] Lustre: Skipped 65 previous similar messages [ 1521.439289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.441460] Lustre: Skipped 7 previous similar messages [ 1523.288953] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1525.005323] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1525.657626] LustreError: 98948:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1525.659083] LustreError: 98948:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1525.712830] Lustre: server umount lustre-OST0000 complete [ 1528.336335] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 12:40:27 (1752511227) [ 1541.922575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.087731] LustreError: 99792:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.093468] LustreError: 99792:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1543.409108] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1546.419460] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1548.429306] LustreError: 99824:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1548.433410] LustreError: 99824:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1548.436971] LustreError: 99824:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1548.568519] Lustre: server umount lustre-MDT0000 complete [ 1553.682368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1559.343026] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 1562.427191] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1565.668137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1568.003602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1570.245578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1572.506703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.810203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.834064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.925329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1574.938500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1574.978423] Lustre: lustre-MDT0000: new disk, initializing [ 1575.009764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1576.392915] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1580.655162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1580.679286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1580.712206] Lustre: 103833:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1580.819958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1582.107138] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1585.134106] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1586.290456] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1588.449852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1588.470474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1590.324220] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1590.880106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1593.280485] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1594.473369] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1594.559618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1595.784788] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1595.864566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1600.997989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.998081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.998766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1600.998772] Lustre: Skipped 17 previous similar messages [ 1601.000960] LustreError: Skipped 2 previous similar messages [ 1601.005081] Lustre: Skipped 6 previous similar messages [ 1602.595257] Lustre: server umount lustre-OST0000 complete [ 1618.399092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1618.450772] Lustre: server umount lustre-MDT0000 complete [ 1619.619156] LustreError: 103824:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511319 with bad export cookie 6318653204906734404 [ 1619.621692] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1619.623175] LustreError: 103824:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1619.752178] Lustre: server umount lustre-MDT0001 complete [ 1622.491625] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 12:42:01 (1752511321) [ 1626.445569] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1627.535611] Key type lgssc unregistered [ 1627.672394] LNet: 107870:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.709312] LNet: Removed LNI 192.168.203.134@tcp [ 1629.010353] Key type .llcrypt unregistered [ 1629.011489] Key type ._llcrypt unregistered [ 1634.571752] Key type ._llcrypt registered [ 1634.573283] Key type .llcrypt registered [ 1634.615170] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 1640.743886] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1641.046270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.054825] alg: No test for adler32 (adler32-zlib) [ 1641.911543] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 1641.988851] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1641.991298] LNet: Accept secure, port 988 [ 1643.575183] Key type lgssc registered [ 1643.918673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.955769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1649.239252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1651.402756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1653.639470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1655.506407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1657.830531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1657.851088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.857092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.953305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1658.968726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1659.011953] Lustre: lustre-MDT0000: new disk, initializing [ 1659.040127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.048279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1660.456337] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1664.496082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1664.517879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1664.601248] Lustre: lustre-OST0000: new disk, initializing [ 1664.602672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1664.604119] Lustre: Skipped 1 previous similar message [ 1664.623733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.916777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1665.919990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1665.929718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1666.295261] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1670.462694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1670.486653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1670.538448] Lustre: lustre-OST0001: new disk, initializing [ 1670.540319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1670.566920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1672.516232] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1675.270977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1675.274504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1675.283812] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1680.352404] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.353063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1680.357248] Lustre: Skipped 1 previous similar message [ 1682.682792] LustreError: 113383:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1682.769937] Lustre: server umount lustre-MDT0000 complete [ 1685.071463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1686.660215] LustreError: 111159:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511386 with bad export cookie 11705577169403096991 [ 1686.662350] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.664489] LustreError: 111159:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1686.684869] LustreError: 113979:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1686.687329] LustreError: 113979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1686.705483] Lustre: server umount lustre-OST0000 complete [ 1687.933168] LustreError: 114179:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1687.934843] LustreError: 114179:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1687.998196] Lustre: server umount lustre-OST0001 complete [ 1690.773695] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 12:43:09 (1752511389) [ 1693.982663] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1697.863398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1698.320439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1701.796436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1702.478976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1703.181944] hrtimer: interrupt took 2089071 ns [ 1707.384769] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 12:43:25 (1752511405) [ 1716.673618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1718.180146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1719.551474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1720.994954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1721.693565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1723.082923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1735.986962] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 1742.264287] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1747.280214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1750.613757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1753.880154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1756.959915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1760.426141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1760.465082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.591702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1760.611809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1760.654338] Lustre: lustre-MDT0000: new disk, initializing [ 1760.694944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.705534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1762.301535] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1767.721401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1767.765460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.816856] Lustre: 119473:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1767.846962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1767.849618] Lustre: Skipped 1 previous similar message [ 1767.890117] Lustre: lustre-MDT0001: new disk, initializing [ 1767.951081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.979508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1767.983487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1769.932110] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1773.563246] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1775.217573] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1778.625743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.661464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1778.783356] Lustre: lustre-OST0000: new disk, initializing [ 1778.785561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1778.832244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.531968] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1785.347288] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.374080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1786.379081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1786.441058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1787.131681] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1787.240483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1789.063172] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1789.166388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1791.460019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.464213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1791.466082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1791.469345] LustreError: Skipped 1 previous similar message [ 1791.474726] Lustre: Skipped 2 previous similar messages [ 1792.046120] LustreError: 122379:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1792.054041] LustreError: 122379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1792.130198] Lustre: server umount lustre-OST0000 complete [ 1796.577025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.581189] Lustre: Skipped 1 previous similar message [ 1796.584283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.697815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.700460] Lustre: Skipped 2 previous similar messages [ 1806.816428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.818957] Lustre: Skipped 1 previous similar message [ 1808.351100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1808.387066] LustreError: 122581:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1808.389783] LustreError: 122581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1808.417923] Lustre: server umount lustre-MDT0000 complete [ 1809.752087] LustreError: 121207:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511509 with bad export cookie 11705577169403098195 [ 1809.754892] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1809.756246] LustreError: 121207:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1809.914630] Lustre: server umount lustre-MDT0001 complete [ 1813.237137] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 12:45:12 (1752511512) [ 1815.858589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.102384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.632629] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1820.978771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1823.002622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1824.911669] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1826.152673] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1829.084820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1831.678052] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1833.782362] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1846.580631] Lustre: *** cfs_fail_loc=159, val=0*** [ 1846.583789] Lustre: Skipped 2 previous similar messages [ 1862.721846] Lustre: lustre-MDT0000: Client 5b6a2d2e-c70b-4a46-9362-3b14a987afed (at 192.168.203.34@tcp) reconnecting [ 1863.432413] Lustre: *** cfs_fail_loc=159, val=0*** [ 1863.434661] Lustre: Skipped 3 previous similar messages [ 1878.594836] Lustre: lustre-MDT0000: Client 5b6a2d2e-c70b-4a46-9362-3b14a987afed (at 192.168.203.34@tcp) reconnecting [ 1882.593974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.595484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.599601] Lustre: Skipped 1 previous similar message [ 1882.604508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1882.607407] Lustre: Skipped 1 previous similar message [ 1886.192626] LustreError: 125945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1886.208136] LustreError: 125945:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1886.290675] Lustre: server umount lustre-OST0000 complete [ 1892.835887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.845183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.849477] LustreError: Skipped 1 previous similar message [ 1894.204596] Lustre: server umount lustre-MDT0000 complete [ 1895.984563] LustreError: 123433:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511596 with bad export cookie 11705577169403099063 [ 1895.987804] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1895.990133] LustreError: 123433:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1896.183316] Lustre: server umount lustre-MDT0001 complete [ 1902.214323] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1903.635664] Key type lgssc unregistered [ 1903.810671] LNet: 126873:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.869413] LNet: Removed LNI 192.168.203.134@tcp [ 1905.328269] Key type .llcrypt unregistered [ 1905.329931] Key type ._llcrypt unregistered [ 1915.919717] Key type ._llcrypt registered [ 1915.921500] Key type .llcrypt registered [ 1915.969043] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1916.347634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.379675] alg: No test for adler32 (adler32-zlib) [ 1917.285056] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 1917.386957] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1917.389555] LNet: Accept secure, port 988 [ 1918.984526] Key type lgssc registered [ 1919.446748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1922.946727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1922.956437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.185490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.122433] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1930.143480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1930.342405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.079198] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1934.618361] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1936.136824] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1938.846637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1938.995406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.312343] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1943.458150] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1949.222846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1954.606922] Lustre: *** cfs_fail_loc=159, val=0*** [ 1970.762340] Lustre: lustre-MDT0000: Client 77445fd7-ec85-4a6d-8fe8-5870914920cd (at 192.168.203.34@tcp) reconnecting [ 1971.844723] Lustre: *** cfs_fail_loc=159, val=0*** [ 1971.847704] Lustre: Skipped 1 previous similar message [ 1988.161287] Lustre: lustre-MDT0001: Client 77445fd7-ec85-4a6d-8fe8-5870914920cd (at 192.168.203.34@tcp) reconnecting [ 1988.729797] Lustre: *** cfs_fail_loc=159, val=0*** [ 1988.732022] Lustre: Skipped 4 previous similar messages [ 2005.055616] Lustre: lustre-MDT0001: Client 77445fd7-ec85-4a6d-8fe8-5870914920cd (at 192.168.203.34@tcp) reconnecting [ 2019.716762] Lustre: 130943:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2020.743406] Lustre: *** cfs_fail_loc=159, val=0*** [ 2020.747271] Lustre: Skipped 2 previous similar messages [ 2037.315361] Lustre: lustre-MDT0000: Client dd873e7f-491d-4f8f-a5ce-667d38658d83 (at 192.168.203.34@tcp) reconnecting [ 2037.994712] Lustre: *** cfs_fail_loc=159, val=0*** [ 2037.996643] Lustre: Skipped 13 previous similar messages [ 2054.208116] Lustre: lustre-MDT0000: Client dd873e7f-491d-4f8f-a5ce-667d38658d83 (at 192.168.203.34@tcp) reconnecting [ 2054.825801] Lustre: 131233:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2057.698067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2057.704553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.714814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2059.748063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2059.748316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.753644] LustreError: Skipped 1 previous similar message [ 2059.760169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2061.793876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2061.800104] Lustre: Skipped 1 previous similar message [ 2062.896197] LustreError: 131331:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2062.969745] Lustre: server umount lustre-OST0000 complete [ 2066.912673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.913851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.920909] Lustre: Skipped 1 previous similar message [ 2066.925113] Lustre: Skipped 1 previous similar message [ 2070.918551] LustreError: 131533:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2070.920601] LustreError: 131533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2070.951195] Lustre: server umount lustre-MDT0000 complete [ 2072.032042] LustreError: 131122:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.037720] LustreError: 131122:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2072.449497] LustreError: 130821:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511772 with bad export cookie 18020847451402893880 [ 2072.452187] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.455303] LustreError: 130821:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2072.526766] LustreError: 131736:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2072.529825] LustreError: 131736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2072.628178] Lustre: server umount lustre-MDT0001 complete [ 2078.214245] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2079.738481] Key type lgssc unregistered [ 2079.934173] LNet: 132262:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.997932] LNet: Removed LNI 192.168.203.134@tcp [ 2081.379446] Key type .llcrypt unregistered [ 2081.381310] Key type ._llcrypt unregistered [ 2092.792965] Key type ._llcrypt registered [ 2092.794574] Key type .llcrypt registered [ 2092.839624] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2093.321973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2093.352045] alg: No test for adler32 (adler32-zlib) [ 2094.258983] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 2094.389326] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2094.392241] LNet: Accept secure, port 988 [ 2096.015163] Key type lgssc registered [ 2096.521208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.302646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2100.316468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.572246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.566200] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2107.505437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2107.696988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.524576] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2112.067709] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2113.753545] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2117.176423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2117.435565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.442701] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 2119.991799] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2122.608389] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2126.644560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 2130.107833] Lustre: 135856:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2130.927064] Lustre: 135904:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2131.615232] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 2133.092589] Lustre: 136051:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 2133.468668] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 2133.471712] Lustre: Skipped 1 previous similar message [ 2133.829971] Lustre: 136149:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2137.057589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.059088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.059407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2137.062597] Lustre: Skipped 1 previous similar message [ 2141.112099] LustreError: 136246:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2141.193490] Lustre: server umount lustre-OST0000 complete [ 2143.715267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2143.719701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2143.731091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.734188] Lustre: Skipped 1 previous similar message [ 2147.296144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.296664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.936528] LustreError: 136455:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2148.939145] LustreError: 136455:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2149.007126] Lustre: server umount lustre-MDT0000 complete [ 2150.666190] LustreError: 135954:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511850 with bad export cookie 9042426635637013245 [ 2150.669193] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.672486] LustreError: 135954:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.748289] LustreError: 136659:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2150.751753] LustreError: 136659:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2150.858962] Lustre: server umount lustre-MDT0001 complete [ 2156.138945] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2157.312689] Key type lgssc unregistered [ 2157.464925] LNet: 137184:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.502834] LNet: Removed LNI 192.168.203.134@tcp [ 2158.865821] Key type .llcrypt unregistered [ 2158.867040] Key type ._llcrypt unregistered [ 2169.169927] Key type ._llcrypt registered [ 2169.171453] Key type .llcrypt registered [ 2169.213910] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2169.691042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.705383] alg: No test for adler32 (adler32-zlib) [ 2170.604973] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 2170.700287] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2170.703017] LNet: Accept secure, port 988 [ 2172.295204] Key type lgssc registered [ 2172.731364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2175.741221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2175.751543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.946199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.438922] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2181.889037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2182.035515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.594349] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2185.771437] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2187.259552] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2189.926585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2190.103277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.168905] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 2192.288203] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2194.467866] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2201.391422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 2207.593904] Lustre: *** cfs_fail_loc=159, val=0*** [ 2207.596080] Lustre: Skipped 2 previous similar messages [ 2224.190088] Lustre: lustre-MDT0000: Client 95bd7c97-8896-4f17-ad7d-af93d32bef96 (at 192.168.203.34@tcp) reconnecting [ 2227.171133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.171351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2227.172906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2227.175822] Lustre: Skipped 1 previous similar message [ 2231.790576] LustreError: 140929:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2231.856736] Lustre: server umount lustre-OST0000 complete [ 2233.312285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2233.315564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.320421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.324324] Lustre: Skipped 1 previous similar message [ 2237.409971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.410025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.358658] LustreError: 141131:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2239.361932] LustreError: 141131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2239.386886] Lustre: server umount lustre-MDT0000 complete [ 2240.948076] LustreError: 139266:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752511940 with bad export cookie 17976541117207779382 [ 2240.951445] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.952424] LustreError: 139266:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2241.033988] LustreError: 141333:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2241.036714] LustreError: 141333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2241.130464] Lustre: server umount lustre-MDT0001 complete [ 2246.356001] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2247.594689] Key type lgssc unregistered [ 2247.740342] LNet: 141859:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.806771] LNet: Removed LNI 192.168.203.134@tcp [ 2249.187902] Key type .llcrypt unregistered [ 2249.189574] Key type ._llcrypt unregistered [ 2260.037942] Key type ._llcrypt registered [ 2260.039422] Key type .llcrypt registered [ 2260.080680] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2260.438107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2260.510711] alg: No test for adler32 (adler32-zlib) [ 2261.374299] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 2261.467115] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2261.468866] LNet: Accept secure, port 988 [ 2263.047174] Key type lgssc registered [ 2263.415400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2266.299978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2266.313920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.496615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.949255] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2272.071234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2272.239286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2273.801854] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2276.022173] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2277.581463] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2280.313933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2280.487444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.535056] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2282.651861] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2284.760111] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2292.784073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2308.836674] Lustre: lustre-OST0000: evicting dbe6e23e-b627-4f78-8285-40a68817b037 (at 192.168.203.34@tcp) by administrative request [ 2308.844712] Lustre: Skipped 1 previous similar message [ 2333.665884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.666184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.670602] LustreError: Skipped 1 previous similar message [ 2333.677229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.680831] Lustre: Skipped 1 previous similar message [ 2338.785840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.788188] Lustre: Skipped 1 previous similar message [ 2339.504193] LustreError: 146005:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2339.574327] Lustre: server umount lustre-OST0000 complete [ 2343.906843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.907576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.912101] Lustre: Skipped 1 previous similar message [ 2343.917604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.920329] Lustre: Skipped 1 previous similar message [ 2347.276241] LustreError: 146207:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2347.280105] LustreError: 146207:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2347.335278] Lustre: server umount lustre-MDT0000 complete [ 2348.909233] LustreError: 144837:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752512048 with bad export cookie 18136042636470946563 [ 2348.912739] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2348.914881] LustreError: 144837:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2348.991935] LustreError: 146408:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2348.994682] LustreError: 146408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2349.112448] Lustre: server umount lustre-MDT0001 complete [ 2354.454958] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2355.654356] Key type lgssc unregistered [ 2355.805496] LNet: 146934:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.839369] LNet: Removed LNI 192.168.203.134@tcp [ 2357.216832] Key type .llcrypt unregistered [ 2357.219986] Key type ._llcrypt unregistered [ 2388.586910] Key type ._llcrypt registered [ 2388.588450] Key type .llcrypt registered [ 2388.638274] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2389.070098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2389.103073] alg: No test for adler32 (adler32-zlib) [ 2389.975377] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 2390.087253] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2390.089456] LNet: Accept secure, port 988 [ 2391.695175] Key type lgssc registered [ 2392.114725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2395.406752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2395.418506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.627873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.182221] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2401.808640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2402.004592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.704801] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2406.128313] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2407.777599] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2410.612519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2410.789268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.819782] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2413.348327] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2415.671368] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2421.050436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2426.338747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.339943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2426.346341] Lustre: Skipped 1 previous similar message [ 2426.351958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.246257] LustreError: 151602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2427.321578] Lustre: server umount lustre-OST0000 complete [ 2431.047508] LustreError: 151806:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2431.049963] LustreError: 151806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2431.089142] Lustre: server umount lustre-MDT0000 complete [ 2431.457866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.458433] LustreError: 149363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.463758] Lustre: Skipped 1 previous similar message [ 2431.469927] LustreError: 149363:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2432.565456] LustreError: 149346:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752512132 with bad export cookie 15305595033100723796 [ 2432.568103] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.570871] LustreError: 149346:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2432.641395] LustreError: 152011:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2432.644414] LustreError: 152011:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2432.744516] Lustre: server umount lustre-MDT0001 complete [ 2438.085282] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2439.369250] Key type lgssc unregistered [ 2439.540295] LNet: 152538:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.614313] LNet: Removed LNI 192.168.203.134@tcp [ 2440.975309] Key type .llcrypt unregistered [ 2440.976911] Key type ._llcrypt unregistered [ 2451.903652] Key type ._llcrypt registered [ 2451.905899] Key type .llcrypt registered [ 2451.958355] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2452.432504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2452.443718] alg: No test for adler32 (adler32-zlib) [ 2453.323184] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 2453.434863] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2453.439286] LNet: Accept secure, port 988 [ 2455.031153] Key type lgssc registered [ 2455.458062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.376926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2461.494989] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 12:56:00 (1752512160) [ 2468.548777] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2472.814766] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2477.214547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.946049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2482.615344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2485.367379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.313077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.336776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.345223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.445055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2489.463907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2489.502783] Lustre: lustre-MDT0000: new disk, initializing [ 2489.539925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.549823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2491.119295] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2496.089695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.126511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2496.165618] Lustre: 156702:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2496.192143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2496.194923] Lustre: Skipped 1 previous similar message [ 2496.232763] Lustre: lustre-MDT0001: new disk, initializing [ 2496.280183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.298518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2496.303559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2497.869759] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2501.612718] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2503.146793] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2506.188816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2506.218885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2506.349451] Lustre: lustre-OST0000: new disk, initializing [ 2506.352159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2506.397663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.859041] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2512.507594] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2513.923990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2513.928935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2513.972262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2524.130738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2524.130846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.133731] LustreError: Skipped 1 previous similar message [ 2524.141546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.143960] Lustre: Skipped 1 previous similar message [ 2527.926210] LustreError: 159272:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2528.014515] Lustre: server umount lustre-OST0000 complete [ 2532.319991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2532.325105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.331930] Lustre: Skipped 1 previous similar message [ 2532.336904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.368382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.368894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.811594] LustreError: 159473:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2535.814287] LustreError: 159473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2535.853291] Lustre: server umount lustre-MDT0000 complete [ 2537.591418] LustreError: 156694:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752512237 with bad export cookie 2503784343022606115 [ 2537.596131] LustreError: 156694:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2537.596723] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.674043] LustreError: 159676:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2537.677371] LustreError: 159676:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2537.782557] Lustre: server umount lustre-MDT0001 complete [ 2543.567687] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2544.884395] Key type lgssc unregistered [ 2545.058076] LNet: 160202:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.085899] LNet: Removed LNI 192.168.203.134@tcp [ 2546.544076] Key type .llcrypt unregistered [ 2546.545727] Key type ._llcrypt unregistered [ 2557.652582] Key type ._llcrypt registered [ 2557.654119] Key type .llcrypt registered [ 2557.694862] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2558.142820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.193926] alg: No test for adler32 (adler32-zlib) [ 2559.086749] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 2559.204068] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2559.207232] LNet: Accept secure, port 988 [ 2560.808209] Key type lgssc registered [ 2561.255401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.557236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.566037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.775869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.347993] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2571.155049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2571.304877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.873997] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2575.163419] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2576.580589] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2579.409526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2579.566636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.640343] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2583.753855] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2590.826331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2596.865809] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2596.873989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.882429] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2596.887047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.891674] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2597.280189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.676359] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2597.679128] Lustre: Skipped 1 previous similar message [ 2597.688508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.694263] Lustre: Skipped 1 previous similar message [ 2597.697156] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2597.700487] Lustre: Skipped 1 previous similar message [ 2597.702873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.709047] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2597.712546] Lustre: Skipped 1 previous similar message [ 2598.101647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.442867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.793035] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2598.795879] Lustre: Skipped 2 previous similar messages [ 2598.803313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.808946] Lustre: Skipped 2 previous similar messages [ 2598.811248] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2598.813686] Lustre: Skipped 2 previous similar messages [ 2598.815661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.820175] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2598.822721] Lustre: Skipped 2 previous similar messages [ 2599.143159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.474260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.834740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.186718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.539890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.864400] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2600.867686] Lustre: Skipped 5 previous similar messages [ 2600.875582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.880482] Lustre: Skipped 5 previous similar messages [ 2600.882531] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2600.885219] Lustre: Skipped 5 previous similar messages [ 2600.887178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.892553] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2600.895477] Lustre: Skipped 5 previous similar messages [ 2601.258475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.586776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.914545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.271195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.649547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.006844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.380952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.721773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.078401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.430801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.800639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.146433] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2605.149319] Lustre: Skipped 11 previous similar messages [ 2605.156137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.161347] Lustre: Skipped 11 previous similar messages [ 2605.163878] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2605.166515] Lustre: Skipped 11 previous similar messages [ 2605.168138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.172734] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2605.177422] Lustre: Skipped 11 previous similar messages [ 2605.507582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.841563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.182674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.516329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.841984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.197037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.570359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.912314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.245075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.569887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.905362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.221777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.574988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.915618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.256933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.597571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.978942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.310562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.639954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.973385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.332904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.669361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.038553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.377659] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2613.381159] Lustre: Skipped 23 previous similar messages [ 2613.388674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.393548] Lustre: Skipped 23 previous similar messages [ 2613.395907] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2613.398167] Lustre: Skipped 23 previous similar messages [ 2613.401652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.405925] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2613.408737] Lustre: Skipped 23 previous similar messages [ 2613.724893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.052498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.411985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.753611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.105502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.475588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.810534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.147859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.492114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.835079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.150520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.513628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.849852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.171533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.508283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.806405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.132488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.443538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.771471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.101243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.434281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.753667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.083027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.423840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.770434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.088525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.423875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.759595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.097269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.396535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.686289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2623.996068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.322561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.617068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2624.921408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.245525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.579968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2625.918797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.241314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.575509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2626.912754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.238423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.610812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2627.964771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.278103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.596213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2628.917582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.244934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.546397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2629.550022] Lustre: Skipped 48 previous similar messages [ 2629.555222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.561234] Lustre: Skipped 48 previous similar messages [ 2629.563948] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2629.567746] Lustre: Skipped 48 previous similar messages [ 2629.570312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2629.574775] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2629.577878] Lustre: Skipped 48 previous similar messages [ 2629.878030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.192945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.534254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2630.878241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.209334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.532874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2631.837615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.144601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.486496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2632.799883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.120952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.433907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2633.779391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.113853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.426470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2634.749587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.094174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.442141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2635.822883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.225573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.587181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2636.978619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.406513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2637.838886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.240031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2638.653192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.010867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.374565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2639.784514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.153967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.515857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2640.903359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.305713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2641.674146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.041441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.452906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2642.841292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.217744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.585354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2643.967406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.403133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2644.756228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.139039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.574203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2645.953287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.344187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2646.742789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.121876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2647.562245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.001051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.459030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.879393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.286144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2649.692422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.112123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.513288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2650.912313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2651.303112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2651.734962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2652.152982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2652.582253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2652.996227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2653.339320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2653.693742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.028256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.356237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2654.688777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2655.015705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2655.356665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2655.684248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2656.068307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2656.475518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2656.864343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2657.230819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2657.616788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2658.007379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2658.397107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2658.745481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2659.092400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2659.419614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2659.757129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.085816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.413676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2660.753516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.091912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.419601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.751503] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2661.756418] Lustre: Skipped 86 previous similar messages [ 2661.763524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.767945] Lustre: Skipped 86 previous similar messages [ 2661.770664] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2661.772779] Lustre: Skipped 86 previous similar messages [ 2661.774625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2661.778111] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2661.781731] Lustre: Skipped 86 previous similar messages [ 2662.118632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2662.528352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2662.891209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.247449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.592045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.947313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2664.300178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2664.674727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2665.038494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2665.431517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2665.818120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2666.196793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2666.531198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2666.861806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.201674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.551448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2667.911892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2668.285681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2668.625794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2669.005918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2669.350648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2669.696722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2670.074459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2670.418214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2670.749266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2671.081816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2671.440671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2671.789337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2672.152567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2672.503838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2672.843916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2673.228710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2673.579531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2673.934687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2674.269080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2674.588924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2674.957707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.292196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.634269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2675.974578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2676.318596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2676.651346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2676.993519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2677.386640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2677.787146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2678.138364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2678.509630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2678.853242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2679.194087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2679.535921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2679.873976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2680.215841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2680.569124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2680.912773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2681.254147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2681.671925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2682.029614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2682.388684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2682.751878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.146206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.544427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2683.929622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2684.314731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2684.671147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2685.012413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2685.343513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2685.718150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2686.061316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2686.383548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2686.726583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2687.076925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2687.417470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2687.799445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2688.203938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2688.540984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2688.892558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2689.294641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2689.673994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2690.041078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2690.407404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2690.757426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2691.109219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2691.541362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2691.878335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2692.249471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2692.592517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2692.954369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2693.301949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2693.653677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2693.991790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2694.322952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2694.682331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2695.018635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2695.371909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2695.697460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2696.049518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2696.392625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2696.752262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2697.134692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2697.466900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2697.806309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.166767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.502519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2698.841606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2699.172447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2699.512270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2699.855655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2700.216766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2700.551554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2700.887684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2701.214207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2701.540911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2701.887899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.215221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.566180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2702.933610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2703.336317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2703.724229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.066971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.403290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2704.738974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2705.105425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2705.460978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2705.856305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2706.188369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2706.492477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2706.820244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2707.155047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2707.502646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2707.822787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2708.150594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2708.484439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2708.834056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.151847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.483111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2709.798585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.145445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.472488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2710.831296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.204256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.559716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2711.896281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.233518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.589542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2712.936442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.269478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.614801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2713.956614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.303112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.654913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.994968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.345679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2715.707814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.050218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.416216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2716.801515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.165507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.530706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2717.850928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.169884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.513122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2718.833156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.158903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.496764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2719.838330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.163384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.513465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.856548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.207055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.553792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.903435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.320249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.700411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.089985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.453613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.821140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.224943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.605423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2724.972473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.346073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2725.714906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.079994] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2726.082012] Lustre: Skipped 181 previous similar messages [ 2726.087552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.092526] Lustre: Skipped 181 previous similar messages [ 2726.094964] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2726.097252] Lustre: Skipped 181 previous similar messages [ 2726.098646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.103472] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2726.106278] Lustre: Skipped 181 previous similar messages [ 2726.456804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2726.840656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.182138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.517548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2727.854335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.194808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.512611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.823313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.138290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.465178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2729.783264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.095864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.410306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.727769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.044126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.362467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2731.687802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.024799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.349585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2732.676028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.007903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.362147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2733.680446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.007894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.361218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2734.715205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.054203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.384579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2735.734932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.079801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.425932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2736.815765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.259769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2737.667181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.065639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.474151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.840635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.230740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.604783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2739.982859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.331213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2740.660379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.019069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.357077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2741.728252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.118838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.478518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2742.850844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.219386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.606310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2743.981624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.376419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2744.855900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.347850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2745.776447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.221090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2746.670650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.253404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2747.803147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.299384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2748.794346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2749.382181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.026279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2750.659291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2751.324255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.080953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2752.848461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2753.549492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2754.199625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.067669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2755.781386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2756.593984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.470085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2758.316694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2759.196117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.141081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2760.983734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2762.109185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.076736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2763.873613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2764.575951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.268616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2765.978895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2766.714283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2767.434958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.176869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2768.942771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2769.630443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2770.314847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.013801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2771.697889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2772.455842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.251133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2773.891320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.583535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2775.310775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2776.119839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.041769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2777.716947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2778.485820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.217775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2779.937588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2780.632321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.289762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2781.879675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.586554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.180888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2783.764825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.290623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2784.786484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.292478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2785.804908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.339118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.824083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.356753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2787.802130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.261111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2788.662105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.086950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.482447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2789.908727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.312133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2790.719320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.133333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.521359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2791.989909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.378442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2792.788258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.177478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.563367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2793.949877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.336836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2794.701829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.062249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.480426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2795.830093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.204738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.552926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2796.946540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.331772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.715130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.096093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.435264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2798.817131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.176808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.550082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2799.928799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.307527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2800.672284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.046882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.471643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2801.879700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.309493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2802.705140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.110230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.497109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2803.897397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.276474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2804.652540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.042757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.437134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2805.868754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.204701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.569858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2806.941911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.332086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2807.783408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.134646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.488393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2808.868474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.263734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.622518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2809.978990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.318287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2810.716996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.089176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.433548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2811.932380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.320447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2812.716100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.072118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.402794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2813.754749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.134059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.530965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2814.930416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.290651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2815.644809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.047380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.451588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2816.851116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.195636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.573866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2817.988314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.402615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2818.801753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.203230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2819.624992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.051175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.419219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2820.805302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.228534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.606233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2821.984111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.369938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2822.832415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.214101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.579221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2823.982130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.333865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2824.741264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.155070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.519896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2825.940739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.345241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2826.749351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.168900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.558547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2827.998977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.413833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2828.794790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.245639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2829.672304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.073840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.544280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2830.940934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.401157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2831.841620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.287341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2832.791440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.230164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2833.652343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.124579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.569062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2834.982647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.413741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2835.829403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.200799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.552149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2836.924136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.342069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2837.792167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.188727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.572091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2838.971572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.365934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2839.748893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.167377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.591481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2840.975388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.357700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2841.724572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.084271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.474682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2842.829664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.205497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.587652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2843.962372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.415647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2844.787413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.170217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.577629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2845.977457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.353088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2846.749779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.122279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.504599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2847.890652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.242133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2848.630676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.013739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.379888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2849.788066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.158566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.576323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2850.970289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.309725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2851.660876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.003596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.392704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2852.814558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.244536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2853.710892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.202234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2854.205278] Lustre: Skipped 283 previous similar messages [ 2854.214680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.221503] Lustre: Skipped 283 previous similar messages [ 2854.225684] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2854.228450] Lustre: Skipped 283 previous similar messages [ 2854.230660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2854.237657] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 2854.241067] Lustre: Skipped 283 previous similar messages [ 2854.670664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.066109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.427901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2855.913421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.341975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2856.729974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.148391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.540312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2857.947803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.278377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.623214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2858.958537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.304628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.639357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2859.962947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.300385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2860.656706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.007614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.324533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2861.662818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.031683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.374342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2862.709979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.092855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.461978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.823995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.169984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.529195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2864.891731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.233110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.566192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2865.920796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.256687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.579768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2866.904844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.240583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.564694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2867.895668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.228120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.558695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2868.934139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.288557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.631784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2869.954450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.273448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.598050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2870.959942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.300393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.633624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2871.982841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.310493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.616063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2872.938272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.267992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.608110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2873.933643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.267686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.614477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2874.964112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.304324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2875.684989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.026372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.367929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.728643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.115892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.500331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2877.885267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.282125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2878.676528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.120455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.527390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2879.910256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.309817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2880.735817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.223038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.691686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.046413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.420764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2882.816220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.171437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.518681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2883.892100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.234488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.556590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2884.909475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.256590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2885.627977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.075128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.473136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2886.872792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.278310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.644698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2887.997717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.367579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2888.777202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.171940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.558403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2889.968399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.400791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2890.785789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.281809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2891.699238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.112144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2892.609686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.105362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2893.618409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.014950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.444488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2894.985844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.409853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2895.869901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.312163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2896.732163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.139127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.552418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2897.934370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.324038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2898.723129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.115194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.525022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2899.981487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.354365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2900.716417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.081675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.465087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2901.853554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.264523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2902.664171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.042638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.398370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2903.791128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.205783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2904.618759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.024674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.416442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2905.801191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.214486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2906.622889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.055841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.456398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2907.853498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.235863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2908.589217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.008735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.358298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2909.722257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.111493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.481668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2910.835655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.188891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.546257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2911.907405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.270213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.622724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2912.967696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.331511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2913.700512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.067425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.452046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2914.845803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.226587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.601881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2915.977928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.357981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2916.763756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.160700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.546094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2917.963528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.375351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2918.764398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.144180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.482903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2919.821396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.142912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.450910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2920.799532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.124340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.458632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2921.813124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.163192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.530022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2922.863561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.168356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.494778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2923.818652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.135524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.446374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2924.774927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.077441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.399745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2925.718352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.030618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.339916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.653182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2926.968430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.305945] LustreError: 161582:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2927.309963] LustreError: 161582:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2927.316037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.648079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2927.952422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.270597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.604473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2928.922425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.245115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.556035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2929.911114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.260254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.603752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2930.957201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.310947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.639704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2931.984553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.379053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2932.729172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.109774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.437270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2933.772579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.112217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.436049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2934.809215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.155212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.478203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2935.811916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.136044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.452814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2936.767650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.072606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.397410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2937.727872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.036915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.355100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2938.683319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.011749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.347804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.656218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2939.974519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.289654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.613161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.930758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.281960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.635356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2941.973056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.332750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2942.682263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.034323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.424963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2943.849709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.235869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.613768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2944.975078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.363534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2945.756173] LustreError: 161582:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2945.760440] LustreError: 161582:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2945.761470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.144873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.521127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2946.910625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.261154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.633456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2947.997687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.371276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2948.729502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.123866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.481444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2949.832978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.184411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.540243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2950.871242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.205515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.580035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2951.905817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.228475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.564067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2952.928877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.280712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.620298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2953.946838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.283098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.612237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2954.959739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.314235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.652891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.991368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.356210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2956.716948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.077209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.455801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2957.807615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.168801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.536479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2958.882595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.232593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.596300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2959.960569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.332746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2960.730218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.102904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.480359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2961.862269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.231594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2962.617764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.004383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.360861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2963.707257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.067373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.427385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2964.775812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.141295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.460636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2965.810479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.160407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.546613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2966.915467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.319892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2967.745262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.149356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.554048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2968.958874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.345223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2969.706486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.057190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.403366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2970.792593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.192207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.556166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2971.911818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.272371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.619250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2972.975110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.347971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2973.737717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.139183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.513795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2974.871352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.238815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.600682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2975.961295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.334663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2976.720427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.075602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.441495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2977.798360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.192494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.554746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.936453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.337988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2979.736290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.163379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2980.562271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.028092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.463978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2981.860318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.304875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2982.763389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3125.306072] LustreError: 163703:0:(ofd_dev.c:1776:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 3125.310442] LustreError: 161582:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 3190.752197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.756063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.762130] Lustre: Skipped 350 previous similar messages [ 3190.765049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3191.263872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.266422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.128106] LustreError: 215322:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3192.233091] Lustre: server umount lustre-OST0000 complete [ 3194.337358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.339877] Lustre: Skipped 1 previous similar message [ 3199.456226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.458853] Lustre: Skipped 1 previous similar message [ 3199.616392] LustreError: 215523:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3199.619144] LustreError: 215523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3199.720359] Lustre: server umount lustre-MDT0000 complete [ 3200.978765] LustreError: 161544:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752512901 with bad export cookie 13920465573049609205 [ 3200.983742] LustreError: 161544:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3200.983859] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.048041] LustreError: 215724:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3201.049836] LustreError: 215724:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3201.127415] Lustre: server umount lustre-MDT0001 complete [ 3205.888193] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3207.004255] Key type lgssc unregistered [ 3207.147603] LNet: 216250:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3208.165310] LNet: Removed LNI 192.168.203.134@tcp [ 3208.477920] Key type .llcrypt unregistered [ 3208.479186] Key type ._llcrypt unregistered [ 3217.401397] Key type ._llcrypt registered [ 3217.402654] Key type .llcrypt registered [ 3217.449478] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3217.736280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3217.746898] alg: No test for adler32 (adler32-zlib) [ 3218.589529] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 3218.661988] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3218.664190] LNet: Accept secure, port 988 [ 3220.239174] Key type lgssc registered [ 3220.605121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3223.360458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3223.367667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.532900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.911454] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3228.883914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3229.002641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3230.208177] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3231.965253] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3233.144447] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3235.328907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.441764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.301526] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3239.166798] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3249.766725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 3256.677682] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3257.372569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.377067] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3257.378832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3257.381688] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.134@tcp (at 0@lo) [ 3257.381720] LustreError: 217630:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 3259.958070] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 3260.384748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.386362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.392294] Lustre: Skipped 1 previous similar message [ 3264.044070] LustreError: 220012:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3264.100507] Lustre: server umount lustre-OST0000 complete [ 3277.463096] LustreError: 217630:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 3291.924132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3292.039731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.725645] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3295.396045] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3300.261037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3300.264064] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3300.264094] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 3300.267724] Lustre: Skipped 1 previous similar message [ 3301.030753] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 3302.218102] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8be104dfa800.ost_server_uuid 50 [ 3302.731814] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8be104dfa800.ost_server_uuid in FULL state after 0 sec [ 3305.444976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.449565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.449813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.453962] Lustre: Skipped 3 previous similar messages [ 3309.737107] LustreError: 221327:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3309.738844] LustreError: 221327:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3309.797776] Lustre: server umount lustre-OST0000 complete [ 3311.071833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3311.075901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.081134] Lustre: Skipped 1 previous similar message [ 3311.082917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.680197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.680884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.799965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.802610] Lustre: Skipped 2 previous similar messages [ 3325.407117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3325.434443] LustreError: 221529:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3325.435908] LustreError: 221529:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3325.464718] Lustre: server umount lustre-MDT0000 complete [ 3325.920178] LustreError: 217611:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.926344] LustreError: 217611:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3326.744223] LustreError: 217592:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513026 with bad export cookie 246720297516509968 [ 3326.748557] LustreError: 217592:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3326.749790] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.914511] Lustre: server umount lustre-MDT0001 complete [ 3331.767131] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3332.862915] Key type lgssc unregistered [ 3332.996329] LNet: 222257:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.053401] LNet: Removed LNI 192.168.203.134@tcp [ 3334.413966] Key type .llcrypt unregistered [ 3334.415996] Key type ._llcrypt unregistered [ 3346.738435] Key type ._llcrypt registered [ 3346.740032] Key type .llcrypt registered [ 3346.785293] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3347.162631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.211090] alg: No test for adler32 (adler32-zlib) [ 3348.095448] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 3348.196881] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3348.199471] LNet: Accept secure, port 988 [ 3349.783174] Key type lgssc registered [ 3350.159880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3353.616268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3353.624615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.819435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.226085] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3358.893699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.002420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3360.235205] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3361.138778] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3363.297659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3363.403970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.456378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 3365.725289] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3368.381613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.403136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3368.446616] Lustre: lustre-OST0001: new disk, initializing [ 3368.448366] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3368.484812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.693144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3368.697834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3368.747145] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3370.344692] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3375.631308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3380.168623] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3395.040095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3395.043774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.048349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.554694] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.556049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3395.558521] Lustre: Skipped 2 previous similar messages [ 3395.560987] Lustre: Skipped 1 previous similar message [ 3397.692720] LustreError: 227264:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3397.728123] Lustre: server umount lustre-MDT0000 complete [ 3398.965387] LustreError: 225026:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513099 with bad export cookie 14384669832820654436 [ 3398.970668] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3399.038166] LustreError: 227466:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3399.041284] LustreError: 227466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3399.126807] Lustre: server umount lustre-MDT0001 complete [ 3410.867852] LustreError: 227666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3410.870127] LustreError: 227666:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3410.888090] Lustre: server umount lustre-OST0000 complete [ 3422.321927] LustreError: 227869:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3422.324567] LustreError: 227869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3422.392216] Lustre: server umount lustre-OST0001 complete [ 3423.449959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3423.476880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.849545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3423.880644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3424.235545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3424.265458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3424.620887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3424.652642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3429.013132] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3432.149894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.292654] LustreError: 229267:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.300060] LustreError: 229267:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3432.313299] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.462463] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3436.174027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3437.476774] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3438.438459] Lustre: 230365:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3440.588981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3440.694613] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.697481] Lustre: Skipped 1 previous similar message [ 3442.319932] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3442.725283] LustreError: 230723:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.728532] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49761) [ 3445.026815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3446.919918] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3449.863281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3451.031142] Lustre: 232198:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3461.602812] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3461.603966] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3461.607636] Lustre: Skipped 3 previous similar messages [ 3461.609499] Lustre: Skipped 3 previous similar messages [ 3466.304291] LustreError: 232505:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3466.307345] LustreError: 232505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3466.346928] Lustre: server umount mylustre-MDT0000 complete [ 3466.721350] LustreError: 229263:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.728292] LustreError: 229263:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3467.640172] LustreError: 229989:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513167 with bad export cookie 14384669832820657201 [ 3467.643665] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.645280] LustreError: 229989:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3467.793070] Lustre: server umount mylustre-MDT0001 complete [ 3479.027110] LustreError: 232907:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3479.030320] LustreError: 232907:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3479.048333] Lustre: server umount mylustre-OST0000 complete [ 3490.676420] Lustre: server umount mylustre-OST0001 complete [ 3491.721188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3491.749391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.130513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3492.164938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.546091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3492.573400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.948289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3492.979880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3498.243224] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3502.054048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.223094] LustreError: 234508:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.247245] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.250522] Lustre: Skipped 1 previous similar message [ 3503.531912] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3506.452724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3507.974409] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3508.950679] Lustre: 235608:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3511.368038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3512.551096] LustreError: 235962:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.554643] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49793) [ 3513.663648] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3516.707944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3518.954168] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3522.257655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3523.575747] Lustre: 237440:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3532.259132] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3532.261258] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.263317] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.264973] Lustre: Skipped 2 previous similar messages [ 3532.266770] Lustre: Skipped 2 previous similar messages [ 3536.701802] LustreError: 237688:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3536.703392] LustreError: 237688:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3536.731717] Lustre: server umount tfs-MDT0000 complete [ 3537.378289] LustreError: 235219:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.383528] LustreError: 235219:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 3538.067468] LustreError: 235233:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513238 with bad export cookie 14384669832820658867 [ 3538.072434] LustreError: 235233:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3538.072590] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3538.235795] Lustre: server umount tfs-MDT0001 complete [ 3549.700816] Lustre: server umount tfs-OST0000 complete [ 3561.420129] Lustre: server umount tfs-OST0001 complete [ 3562.565341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3562.590848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.947811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3562.978345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.315909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3563.340216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.717066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3563.753960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3569.164699] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3572.957140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.119182] LustreError: 239694:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.141873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.143857] Lustre: Skipped 3 previous similar messages [ 3574.568414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3577.658721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3579.187122] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3580.243839] Lustre: 240791:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3582.687970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3583.847754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49825) [ 3584.856642] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3588.100714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3590.284307] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3593.688865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3594.929391] Lustre: 242625:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3602.404786] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 13:15:01 (1752513301) [ 3618.790683] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.793902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3618.794562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.794569] Lustre: Skipped 2 previous similar messages [ 3618.795877] Lustre: Skipped 4 previous similar messages [ 3620.415723] LustreError: 242994:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3620.417533] LustreError: 242994:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3620.449767] Lustre: server umount lustre-MDT0000 complete [ 3621.661146] LustreError: 242895:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513321 with bad export cookie 14384669832820660309 [ 3621.663906] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.665258] LustreError: 242895:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3623.904406] LustreError: 242885:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.908474] LustreError: 242885:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3626.912165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3626.914126] Lustre: Skipped 6 previous similar messages [ 3627.841044] Lustre: server umount lustre-MDT0001 complete [ 3643.359118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3643.394700] Lustre: server umount lustre-OST0000 complete [ 3650.818352] Lustre: server umount lustre-OST0001 complete [ 3652.879354] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 3656.035913] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3659.713711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3662.182217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3664.454424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3666.810872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3669.451773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3669.475985] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3669.475985] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3669.475985] [ 3669.486692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3669.584990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3669.599872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3669.644058] Lustre: lustre-MDT0000: new disk, initializing [ 3669.672880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.675049] Lustre: Skipped 3 previous similar messages [ 3669.681733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3671.064931] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3675.535452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3675.561174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3675.599044] Lustre: 245763:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3675.623525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3675.625454] Lustre: Skipped 1 previous similar message [ 3675.663626] Lustre: lustre-MDT0001: new disk, initializing [ 3675.725322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3675.730506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3677.227156] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3681.694958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3681.730690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3681.868913] Lustre: lustre-OST0000: new disk, initializing [ 3681.871919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3683.134155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3683.140716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3683.191802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3684.048198] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3688.483928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3688.514056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3688.589339] Lustre: lustre-OST0001: new disk, initializing [ 3688.592293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3689.746982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3689.751546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3689.787974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3690.747641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3695.272797] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 13:16:34 (1752513394) [ 3698.757730] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 13:16:37 (1752513397) [ 3710.435693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.437279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.441162] Lustre: Skipped 3 previous similar messages [ 3710.446319] Lustre: Skipped 1 previous similar message [ 3716.272114] LustreError: 249161:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3716.274665] LustreError: 249161:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3716.293526] Lustre: server umount lustre-OST0000 complete [ 3720.673252] LustreError: 247291:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.680476] LustreError: 247291:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3723.811782] Lustre: server umount lustre-MDT0000 complete [ 3725.237405] LustreError: 249364:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513425 with bad export cookie 14384669832820661639 [ 3725.242832] LustreError: 249364:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3725.243030] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.490207] Lustre: server umount lustre-MDT0001 complete [ 3736.358403] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3746.362872] Lustre: server umount lustre-OST0001 complete [ 3748.529196] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 3755.263060] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3759.071147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3761.429749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3763.481540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3765.453714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3767.646576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3767.669063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.747967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3767.759725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3767.795558] Lustre: lustre-MDT0000: new disk, initializing [ 3767.826908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3769.119714] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3773.265380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3773.290838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3773.323862] Lustre: 252599:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3773.425848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3774.721969] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3777.851099] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3779.069206] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3781.344877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3781.370451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3783.337690] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3783.478366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3783.516418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3786.453800] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3793.890969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3793.890997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.893534] LustreError: Skipped 1 previous similar message [ 3793.896809] Lustre: Skipped 4 previous similar messages [ 3793.897533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.901467] Lustre: Skipped 8 previous similar messages [ 3794.931194] Lustre: server umount lustre-OST0000 complete [ 3799.010168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3802.331072] Lustre: server umount lustre-MDT0000 complete [ 3803.639275] LustreError: 255226:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513503 with bad export cookie 14384669832820663109 [ 3803.642547] LustreError: 255226:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3803.642756] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.808179] Lustre: server umount lustre-MDT0001 complete [ 3808.913942] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3810.013404] Key type lgssc unregistered [ 3810.153383] LNet: 255954:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3811.173308] LNet: Removed LNI 192.168.203.134@tcp [ 3811.492794] Key type .llcrypt unregistered [ 3811.494615] Key type ._llcrypt unregistered [ 3822.000901] Key type ._llcrypt registered [ 3822.001987] Key type .llcrypt registered [ 3822.048971] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3822.480278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.489544] alg: No test for adler32 (adler32-zlib) [ 3823.366863] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 3823.464703] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3823.466485] LNet: Accept secure, port 988 [ 3825.055190] Key type lgssc registered [ 3825.445193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3828.304197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3828.310893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.488642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3830.837687] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3833.838705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3833.970685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3835.322550] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3837.111663] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3838.329157] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3840.706682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3840.829053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.797037] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3844.798738] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3845.165740] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3849.187025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3849.187034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.194865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.882223] LustreError: 259636:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3851.947056] Lustre: server umount lustre-OST0000 complete [ 3854.305074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.305345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3854.305880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.310305] Lustre: Skipped 1 previous similar message [ 3859.424240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.427296] Lustre: Skipped 2 previous similar messages [ 3859.454863] LustreError: 259845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3859.457144] LustreError: 259845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3859.481177] Lustre: server umount lustre-MDT0000 complete [ 3860.831278] LustreError: 259539:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513560 with bad export cookie 10233263090608247207 [ 3860.834040] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.836097] LustreError: 259539:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3860.903049] LustreError: 260046:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3860.905031] LustreError: 260046:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3860.994552] Lustre: server umount lustre-MDT0001 complete [ 3865.142970] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3866.351428] Key type lgssc unregistered [ 3866.488355] LNet: 260572:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3867.557310] LNet: Removed LNI 192.168.203.134@tcp [ 3867.863182] Key type .llcrypt unregistered [ 3867.864295] Key type ._llcrypt unregistered [ 3874.478788] Key type ._llcrypt registered [ 3874.479867] Key type .llcrypt registered [ 3874.524583] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3874.870316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3874.880462] alg: No test for adler32 (adler32-zlib) [ 3875.741876] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 3875.822628] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3875.824195] LNet: Accept secure, port 988 [ 3877.399156] Key type lgssc registered [ 3877.716669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3880.687205] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3880.693499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3897.119320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3903.263245] LustreError: 261712:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 3933.983203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.941409] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3937.733169] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3938.473749] LustreError: 262370:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3938.550406] Lustre: server umount lustre-OST0000 complete [ 3944.373733] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3945.491617] Key type lgssc unregistered [ 3945.646420] LNet: 263200:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3946.661392] LNet: Removed LNI 192.168.203.134@tcp [ 3946.942872] Key type .llcrypt unregistered [ 3946.943968] Key type ._llcrypt unregistered [ 3961.538680] Key type ._llcrypt registered [ 3961.540045] Key type .llcrypt registered [ 3961.589284] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3962.250595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3962.279933] alg: No test for adler32 (adler32-zlib) [ 3963.157105] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 3963.255705] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3963.258682] LNet: Accept secure, port 988 [ 3964.847192] Key type lgssc registered [ 3965.252315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3969.777477] loop0: detected capacity change from 0 to 204800000 [ 3969.837692] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3969.842417] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3969.967976] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3969.971607] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3969.991820] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3970.013955] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3970.031685] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3970.062774] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3970.089604] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3970.119108] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3970.513829] loop0: detected capacity change from 0 to 204800000 [ 3970.588024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3970.866615] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3971.236068] loop0: detected capacity change from 0 to 204800000 [ 3971.380296] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3971.732719] loop0: detected capacity change from 0 to 204800000 [ 3971.885077] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3972.267777] loop0: detected capacity change from 0 to 204800000 [ 3972.286667] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3973.731393] loop0: detected capacity change from 0 to 204800000 [ 3973.743303] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3975.072119] loop0: detected capacity change from 0 to 204800000 [ 3975.086083] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3976.492349] loop0: detected capacity change from 0 to 204800000 [ 3976.507917] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3977.879771] loop0: detected capacity change from 0 to 204800000 [ 3977.915038] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3977.934221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3977.940520] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.970554] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3977.974536] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3977.980887] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3977.987107] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3977.990720] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3977.994900] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3977.999375] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3978.003701] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3978.010190] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3978.016518] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3978.023415] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3978.029449] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3978.032990] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3978.036712] Lustre: 266563:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3980.567897] LustreError: 266830:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3980.609814] Lustre: server umount MGS complete [ 3980.902377] loop0: detected capacity change from 0 to 204800000 [ 3980.943535] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.096306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3981.111672] Lustre: lustre-MDT0000: Aborting client recovery [ 3981.427573] loop1: detected capacity change from 0 to 204800000 [ 3981.461716] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3981.523298] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3981.556933] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3981.561699] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3981.565734] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3981.569506] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3981.574482] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3981.580356] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3981.586648] Lustre: 267000:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3981.595718] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3981.681231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.703512] Lustre: lustre-MDT0001: Aborting client recovery [ 3982.025942] loop2: detected capacity change from 0 to 204800000 [ 3982.075490] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3982.099273] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3982.114217] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3982.117225] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3982.120194] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3982.123831] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3982.127325] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3982.130418] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3982.133407] Lustre: 267085:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3982.180237] Lustre: Found index 0 for lustre-OST0000, updating log [ 3982.211440] Lustre: lustre-OST0000: Aborting recovery [ 3982.536565] loop3: detected capacity change from 0 to 204800000 [ 3982.582309] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3982.604101] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3982.632167] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3982.637441] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3982.642770] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3982.647296] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3982.649877] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3982.653217] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3982.656418] Lustre: 267172:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3982.691650] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3982.693712] Lustre: Skipped 1 previous similar message [ 3991.842815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3991.842875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3991.843707] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3991.851343] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3991.853717] LustreError: 266943:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 3991.855957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3991.860939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3993.117503] LustreError: 266910:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 269us: rc = -2 [ 4007.392413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4007.392735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.393503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.400643] Lustre: Skipped 2 previous similar messages [ 4010.301682] LustreError: 267524:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4010.303653] LustreError: 267524:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4010.340638] Lustre: server umount lustre-MDT0000 complete [ 4010.639537] LustreError: 266892:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513710 with bad export cookie 3834207083562548430 [ 4010.641023] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4010.642892] LustreError: 266892:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4012.511626] LustreError: 267113:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.511795] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.512352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4012.512357] Lustre: Skipped 3 previous similar messages [ 4012.516136] LustreError: 267113:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4012.519103] Lustre: Skipped 2 previous similar messages [ 4015.519685] LustreError: 267113:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.519940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4015.527207] LustreError: 267113:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4015.535827] Lustre: Skipped 2 previous similar messages [ 4016.824573] LustreError: 267577:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4016.826298] LustreError: 267577:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4016.908537] Lustre: server umount lustre-MDT0001 complete [ 4023.274065] LustreError: 267628:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4023.275904] LustreError: 267628:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4023.302057] Lustre: server umount lustre-OST0000 complete [ 4029.672562] LustreError: 267679:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4029.674494] LustreError: 267679:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4029.743818] Lustre: server umount lustre-OST0001 complete [ 4032.369442] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 13:22:11 (1752513731) [ 4041.777538] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4045.439303] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4048.715061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4050.868622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4052.735812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4054.865753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.991607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4057.012120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.085991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4057.095036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4057.127144] Lustre: lustre-MDT0000: new disk, initializing [ 4057.148673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.154550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4058.314562] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4062.292639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.318101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4062.362781] Lustre: 271204:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4062.395933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4062.397533] Lustre: Skipped 1 previous similar message [ 4062.434640] Lustre: lustre-MDT0001: new disk, initializing [ 4062.475226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4062.489741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4062.493542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4063.810726] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4066.896708] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4068.163440] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4070.371551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4070.392813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4070.486081] Lustre: lustre-OST0000: new disk, initializing [ 4070.488930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4070.520278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.397577] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4073.917231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4073.920325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4073.961695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4075.432327] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4077.782754] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4081.578308] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4088.698434] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4095.218050] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 4135.394077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4135.394560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.395386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.395943] LustreError: Skipped 1 previous similar message [ 4135.398284] Lustre: Skipped 1 previous similar message [ 4140.204839] LustreError: 274747:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4140.207197] LustreError: 274747:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4140.290734] Lustre: server umount lustre-OST0000 complete [ 4144.608889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4144.615777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.626644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.629370] Lustre: Skipped 1 previous similar message [ 4155.871431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4155.875054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.878106] Lustre: Skipped 5 previous similar messages [ 4155.981196] Lustre: server umount lustre-MDT0000 complete [ 4158.526362] LustreError: 271979:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513858 with bad export cookie 3834207083562551139 [ 4158.530970] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.532974] LustreError: 271979:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4158.623898] LustreError: 275153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4158.626264] LustreError: 275153:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4158.811399] Lustre: server umount lustre-MDT0001 complete [ 4164.803658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.845503] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4171.715214] Lustre: server umount MGS complete [ 4179.282522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.480626] LustreError: 276711:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.491638] LustreError: 276723:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4179.526943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.132627] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4184.265886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4185.747749] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4187.855488] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4189.277566] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4191.977825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4194.129271] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4196.136680] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4230.115190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.115904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4230.116695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.120614] Lustre: Skipped 2 previous similar messages [ 4230.129211] LustreError: Skipped 1 previous similar message [ 4234.796075] LustreError: 279372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4234.798681] LustreError: 279372:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4234.867948] Lustre: server umount lustre-OST0000 complete [ 4240.352714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.356978] Lustre: Skipped 1 previous similar message [ 4242.388212] Lustre: server umount lustre-MDT0000 complete [ 4243.663108] LustreError: 278317:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752513943 with bad export cookie 3834207083562552952 [ 4243.666368] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.667687] LustreError: 278317:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4243.810848] Lustre: server umount lustre-MDT0001 complete [ 4248.674909] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4249.839133] Key type lgssc unregistered [ 4249.977390] LNet: 280301:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4251.045348] LNet: Removed LNI 192.168.203.134@tcp [ 4251.345625] Key type .llcrypt unregistered [ 4251.346614] Key type ._llcrypt unregistered [ 4263.623739] Key type ._llcrypt registered [ 4263.625166] Key type .llcrypt registered [ 4263.667201] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4270.260439] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4270.549591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4270.555892] alg: No test for adler32 (adler32-zlib) [ 4271.409707] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 4271.495165] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4271.497207] LNet: Accept secure, port 988 [ 4273.079196] Key type lgssc registered [ 4273.445579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4276.596442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4279.006533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4281.464451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4283.948390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4286.785054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4286.816688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4286.829330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.932056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4287.950128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4287.988218] Lustre: lustre-MDT0000: new disk, initializing [ 4288.033773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.046527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4289.683308] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4294.953441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4294.984569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4295.029789] Lustre: 284189:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4295.060139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4295.062556] Lustre: Skipped 1 previous similar message [ 4295.092241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4295.108231] Lustre: lustre-MDT0001: new disk, initializing [ 4295.154718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4295.173166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4295.179806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4296.851641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4300.697917] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4302.429431] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4305.578236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4305.609894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4305.725049] Lustre: lustre-OST0000: new disk, initializing [ 4305.728202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4305.767798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.153414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4311.062462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4311.074901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4311.131405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4312.437735] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4317.984113] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 4323.604526] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 4327.737174] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 4372.448149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.455259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.478143] LustreError: Skipped 1 previous similar message [ 4372.499797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4377.589408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4377.592367] Lustre: Skipped 2 previous similar messages [ 4378.569331] LustreError: 287487:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4378.992923] Lustre: server umount lustre-OST0000 complete [ 4387.297811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4387.319132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.346529] Lustre: Skipped 1 previous similar message [ 4387.358998] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.772209] LustreError: 287689:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4390.793797] LustreError: 287689:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4391.142882] Lustre: server umount lustre-MDT0000 complete [ 4396.278756] LustreError: 284964:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752514096 with bad export cookie 8362543764076693030 [ 4396.286980] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.297770] LustreError: 284964:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4396.683345] LustreError: 287892:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4396.707327] LustreError: 287892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4397.290218] Lustre: server umount lustre-MDT0001 complete [ 4408.855828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.444489] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4419.874359] LustreError: 288830:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4419.893690] LustreError: 288830:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4419.989594] Lustre: server umount MGS complete [ 4431.484646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.916609] LustreError: 289453:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.935378] LustreError: 289453:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4431.944350] LustreError: 289465:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4431.997328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.992712] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4437.474726] LustreError: 289453:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.575801] LustreError: 289453:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.398629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4444.800620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4448.572138] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4454.660455] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4458.100331] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4465.127994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4465.492602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.831615] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4476.186986] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4510.695826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4510.702537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.721643] Lustre: Skipped 1 previous similar message [ 4510.728478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4510.737595] Lustre: Skipped 1 previous similar message [ 4512.738957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4512.743983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.272105] LustreError: 292092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4516.288499] LustreError: 292092:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4516.550845] Lustre: server umount lustre-OST0000 complete [ 4521.440204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4521.443544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.478166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.480821] Lustre: Skipped 2 previous similar messages [ 4526.493784] LustreError: 292293:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4526.501571] LustreError: 292293:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4526.660803] Lustre: server umount lustre-MDT0000 complete [ 4528.095982] LustreError: 289453:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.126118] LustreError: 289453:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4529.734087] LustreError: 291938:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752514229 with bad export cookie 8362543764076694871 [ 4529.734796] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4529.747193] LustreError: 291938:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4530.115211] Lustre: server umount lustre-MDT0001 complete [ 4541.153229] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4543.771794] Key type lgssc unregistered [ 4544.030891] LNet: 293021:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4545.063665] LNet: Removed LNI 192.168.203.134@tcp [ 4545.851245] Key type .llcrypt unregistered [ 4545.854167] Key type ._llcrypt unregistered [ 4564.748428] Key type ._llcrypt registered [ 4564.750051] Key type .llcrypt registered [ 4564.882095] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4577.708342] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4578.350569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4578.368543] alg: No test for adler32 (adler32-zlib) [ 4579.510724] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 4579.726756] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4579.733907] LNet: Accept secure, port 988 [ 4581.391201] Key type lgssc registered [ 4582.584700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4591.705852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4598.173900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.623577] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4617.162916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4623.490416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4623.549678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4623.573743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.884529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4624.923415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4624.993957] Lustre: lustre-MDT0000: new disk, initializing [ 4625.055276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.092142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4628.472427] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4638.792072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4638.853753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4639.032987] Lustre: lustre-OST0000: new disk, initializing [ 4639.040742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4639.048540] Lustre: Skipped 1 previous similar message [ 4639.129325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.799996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4639.804987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4639.841762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4645.123475] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5510.113859] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.131560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.167155] LustreError: 298036:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5512.491382] Lustre: server umount lustre-MDT0000 complete [ 5517.223220] LustreError: 296186:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752515217 with bad export cookie 5139683333168073306 [ 5517.232525] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5517.299692] LustreError: 298240:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5517.311556] LustreError: 298240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5517.591958] Lustre: server umount lustre-OST0000 complete [ 5547.516073] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 5556.780154] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5566.248274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.525061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5579.581913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5585.186090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.023512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.090813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.262771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5591.284067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5591.325898] Lustre: lustre-MDT0000: new disk, initializing [ 5591.395413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.416205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5595.033105] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5604.302380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5604.354372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5604.435276] Lustre: 301271:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5604.485853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5604.489535] Lustre: Skipped 1 previous similar message [ 5604.607659] Lustre: lustre-MDT0001: new disk, initializing [ 5604.733656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5604.799768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5604.814351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5608.509080] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5615.546957] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5618.981125] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5625.704626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5625.784857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5625.980440] Lustre: lustre-OST0000: new disk, initializing [ 5625.997784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5626.070932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.975348] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5632.610236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5632.617349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5632.764565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5640.301945] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5644.011168] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5644.195552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5648.035389] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5648.211842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5653.482637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.482783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5653.503589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5653.508669] LustreError: Skipped 1 previous similar message [ 5653.515548] Lustre: Skipped 1 previous similar message [ 5658.597647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5658.600095] Lustre: Skipped 1 previous similar message [ 5663.719558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.725658] Lustre: Skipped 1 previous similar message [ 5664.223242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5664.261137] LustreError: 304175:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5664.270279] LustreError: 304175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5664.489919] Lustre: server umount lustre-OST0000 complete [ 5668.832937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.838415] Lustre: Skipped 1 previous similar message [ 5668.841919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5674.214788] LustreError: 304379:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5674.220582] LustreError: 304379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5674.268377] Lustre: server umount lustre-MDT0000 complete [ 5676.948366] LustreError: 301261:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752515376 with bad export cookie 5139683333171014153 [ 5676.949453] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5676.963672] LustreError: 301261:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5677.400384] Lustre: server umount lustre-MDT0001 complete [ 5679.880226] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 5680.970314] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 13:49:39 (1752515379) [ 5685.660489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.032731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.220613] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5691.366691] LustreError: 305152:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.385285] LustreError: 305152:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5696.484387] LustreError: 305152:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.549261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5696.802178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5699.740380] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5703.864614] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5706.718538] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5711.867131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5712.122159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.755676] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5722.459797] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5729.535751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5729.635068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5729.784624] Lustre: lustre-OST0001: new disk, initializing [ 5729.789065] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5729.906189] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5735.707502] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5739.134951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5739.153981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5739.191050] LustreError: 308100:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5740.257464] LustreError: 308096:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5740.276824] LustreError: 308096:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 5741.280984] LustreError: 308100:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5743.330535] LustreError: 308100:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5743.338928] LustreError: 308100:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 5746.026160] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5746.257083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5747.494462] LustreError: 308100:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5747.518186] LustreError: 308100:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 5749.708964] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8be12e818000.ost_server_uuid 50 [ 5751.611272] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8be12e818000.ost_server_uuid in FULL state after 0 sec [ 5755.684362] LustreError: 308096:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 5755.696305] LustreError: 308096:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 5755.724672] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8be12e818000.ost_server_uuid 50 [ 5757.081815] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8be12e818000.ost_server_uuid in FULL state after 0 sec [ 5758.846292] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5771.841389] Lustre: lustre-OST0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 5771.843994] Lustre: Skipped 3 previous similar messages [ 5774.310378] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5774.312911] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.319325] Lustre: Skipped 1 previous similar message [ 5774.323167] LustreError: Skipped 1 previous similar message [ 5786.079173] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5786.122170] LustreError: 308935:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5786.124187] LustreError: 308935:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5786.222223] Lustre: server umount lustre-OST0001 complete [ 5787.186996] LustreError: 306736:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.705834] LustreError: 306736:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.725096] LustreError: 306736:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5793.252732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5793.264569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.283920] Lustre: Skipped 1 previous similar message [ 5793.297973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5793.300712] Lustre: Skipped 10 previous similar messages [ 5795.809611] LustreError: 308554:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.831000] LustreError: 308554:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5797.250670] LustreError: 309138:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5797.271245] LustreError: 309138:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5797.480236] Lustre: server umount lustre-OST0000 complete [ 5806.677350] Lustre: server umount lustre-MDT0000 complete [ 5809.740764] LustreError: 305133:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752515509 with bad export cookie 5139683333171015035 [ 5809.743957] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5809.748790] LustreError: 305133:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5810.169901] Lustre: server umount lustre-MDT0001 complete [ 5820.426872] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 5823.285792] Key type lgssc unregistered [ 5823.585570] LNet: 310068:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5824.613919] LNet: Removed LNI 192.168.203.134@tcp [ 5825.330804] Key type .llcrypt unregistered [ 5825.332554] Key type ._llcrypt unregistered [ 5846.580424] Key type ._llcrypt registered [ 5846.582103] Key type .llcrypt registered [ 5846.703103] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 5847.531091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5847.606418] alg: No test for adler32 (adler32-zlib) [ 5848.602462] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 5848.779750] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 5848.783956] LNet: Accept secure, port 988 [ 5850.431181] Key type lgssc registered [ 5851.370806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5857.971647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5857.992473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5859.365412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5863.076529] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5868.506974] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5869.542183] LustreError: 311459:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5869.560149] LustreError: 311459:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5874.678751] LustreError: 311459:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.996905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5875.507465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.653474] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5889.131254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5889.602162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5895.186736] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5899.315256] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5900.839083] LustreError: 313002:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.939205] LustreError: 313002:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.954253] LustreError: 313002:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5904.954730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5905.046475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5905.249279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5910.548300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5910.801641] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 5915.545852] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5920.831752] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5921.033199] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6033.375889] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6033.383035] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.408484] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6033.412124] Lustre: Skipped 1 previous similar message [ 6038.498298] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6043.623362] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6043.633631] Lustre: Skipped 2 previous similar messages [ 6046.175347] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6046.220075] LustreError: 314857:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6046.292769] Lustre: server umount lustre-OST0001 complete [ 6048.743226] LustreError: 313003:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.759761] LustreError: 313003:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6053.344095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6053.347421] LustreError: Skipped 1 previous similar message [ 6053.372870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6053.391789] Lustre: Skipped 1 previous similar message [ 6053.398801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6057.714708] LustreError: 315060:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6057.728796] LustreError: 315060:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6057.908606] Lustre: server umount lustre-OST0000 complete [ 6064.096251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.104687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.113579] Lustre: Skipped 2 previous similar messages [ 6064.120790] Lustre: Skipped 1 previous similar message [ 6074.338082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6074.344115] Lustre: Skipped 3 previous similar messages [ 6075.403169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6075.522529] LustreError: 315262:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6075.534607] LustreError: 315262:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6075.600550] Lustre: server umount lustre-MDT0000 complete [ 6079.112961] LustreError: 311441:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752515779 with bad export cookie 14181914099506705358 [ 6079.115786] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.126599] LustreError: 311441:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6079.237181] LustreError: 315465:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6079.245472] LustreError: 315465:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6079.424499] Lustre: server umount lustre-MDT0001 complete [ 6091.060791] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6093.665382] Key type lgssc unregistered [ 6093.961763] LNet: 315992:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6095.021971] LNet: Removed LNI 192.168.203.134@tcp [ 6095.767443] Key type .llcrypt unregistered [ 6095.768987] Key type ._llcrypt unregistered [ 6131.294852] Key type ._llcrypt registered [ 6131.296426] Key type .llcrypt registered [ 6131.361739] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6131.944357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6131.962270] alg: No test for adler32 (adler32-zlib) [ 6132.897818] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 6133.051627] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 6133.053498] LNet: Accept secure, port 988 [ 6134.671156] Key type lgssc registered [ 6135.435859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6143.094695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6143.119163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.569638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.815037] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6154.724539] LustreError: 318252:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.734716] LustreError: 318252:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6154.990336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6155.326571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6158.168280] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6160.014880] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6165.447537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6165.793175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6169.897483] LustreError: 319700:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.917876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6170.424237] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6175.201687] LustreError: 319700:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.573843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6177.862694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.977412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 6182.557747] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6189.985236] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6194.202413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6211.555231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6211.559689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.567767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6212.067813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.072749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6212.087535] Lustre: Skipped 2 previous similar messages [ 6212.094605] Lustre: Skipped 2 previous similar messages [ 6213.425217] LustreError: 321634:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6213.507530] Lustre: server umount lustre-MDT0000 complete [ 6216.277517] LustreError: 318231:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752515916 with bad export cookie 10988384969089669076 [ 6216.280986] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6216.287090] LustreError: 318231:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6216.379409] LustreError: 321837:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6216.381804] LustreError: 321837:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6216.567715] Lustre: server umount lustre-MDT0001 complete [ 6229.825242] LustreError: 322037:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6229.835325] LustreError: 322037:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6229.889584] Lustre: server umount lustre-OST0000 complete [ 6242.051248] LustreError: 322239:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6242.058985] LustreError: 322239:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6242.233619] Lustre: server umount lustre-OST0001 complete [ 6258.571486] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6267.302633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.683129] LustreError: 323409:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.777169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.561381] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6272.999514] LustreError: 323409:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.189843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6280.789327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6284.221112] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6287.505585] Lustre: 324505:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6294.445533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6300.630269] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6305.131160] LustreError: 324861:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.155918] LustreError: 324861:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6309.835198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6310.080209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6310.085428] Lustre: Skipped 1 previous similar message [ 6313.661159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6313.679682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 6315.185861] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6322.643742] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6325.704259] Lustre: 326342:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6334.433334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.447212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.452652] Lustre: Skipped 3 previous similar messages [ 6339.558628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.561392] Lustre: Skipped 2 previous similar messages [ 6339.838394] LustreError: 326447:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6339.841237] LustreError: 326447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6339.924359] Lustre: server umount lustre-MDT0000 complete [ 6343.385568] LustreError: 323388:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752516043 with bad export cookie 10988384969089670777 [ 6343.399681] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6343.403304] LustreError: 323388:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6343.853364] Lustre: server umount lustre-MDT0001 complete [ 6357.984042] LustreError: 326849:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6358.022041] LustreError: 326849:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6358.094080] Lustre: server umount lustre-OST0000 complete [ 6360.031278] Lustre: 317516:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1752516044/real 1752516044] req@ffffa0a979d1f800 x1837646042208768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1752516060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6360.077356] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.110843] Lustre: Skipped 3 previous similar messages [ 6362.520829] Lustre: server umount lustre-OST0001 complete [ 6366.410581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 6368.013746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 6369.439122] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 14:01:07 (1752516067) [ 6390.903631] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6399.094597] loop0: detected capacity change from 0 to 1099511627776 [ 6399.664894] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6399.678994] *** VALIDATE xfs *** [ 6399.691861] XFS (loop0): Mounting V5 Filesystem [ 6399.727066] XFS (loop0): Ending clean mount [ 6400.943961] loop1: detected capacity change from 0 to 18691697672192 [ 6459.040391] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6476.631735] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 6485.231137] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6493.124421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.387750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6502.980739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6507.403857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6512.347465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6512.397730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.562370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6512.590725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6512.645641] Lustre: lustre-MDT0000: new disk, initializing [ 6512.706610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.719037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6515.373752] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6525.202715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6525.273513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6525.403239] Lustre: 331419:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6525.454323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6525.456848] Lustre: Skipped 1 previous similar message [ 6525.550800] Lustre: lustre-MDT0001: new disk, initializing [ 6525.675492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6525.686142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6529.695938] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6537.644836] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6541.232362] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6547.007282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6547.062793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6547.260196] Lustre: lustre-OST0000: new disk, initializing [ 6547.269381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6549.038327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6549.049524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6549.124817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6551.976592] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6558.636446] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6562.605697] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6562.796403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6566.679207] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6566.858321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6569.442102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.443186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.452916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6569.463322] Lustre: Skipped 1 previous similar message [ 6569.471647] Lustre: Skipped 1 previous similar message [ 6574.396152] LustreError: 334327:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6574.398825] LustreError: 334327:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6574.601051] Lustre: server umount lustre-OST0000 complete [ 6579.682191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.684088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6579.692090] Lustre: Skipped 1 previous similar message [ 6579.708020] Lustre: Skipped 1 previous similar message [ 6583.920599] Lustre: server umount lustre-MDT0000 complete [ 6584.802099] LustreError: 331428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.818304] LustreError: 331428:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6586.890695] LustreError: 331411:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752516286 with bad export cookie 10988384969089672331 [ 6586.895764] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6586.900678] LustreError: 331411:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6587.231238] Lustre: server umount lustre-MDT0001 complete [ 6593.784285] XFS (loop0): Unmounting Filesystem [ 6597.680417] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 14:04:55 (1752516295) [ 6603.440222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6604.037324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.042572] Lustre: Skipped 2 previous similar messages [ 6607.303790] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6613.691755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6617.004592] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6620.970268] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6624.014794] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6629.658632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6634.342263] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6639.181889] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6646.243379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6646.248448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.261500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.222162] LustreError: 337869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6652.231212] LustreError: 337869:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6652.388570] Lustre: server umount lustre-OST0000 complete [ 6661.985279] Lustre: server umount lustre-MDT0000 complete [ 6664.676585] LustreError: 335544:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6664.695081] LustreError: 335544:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6665.332776] LustreError: 335528:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752516365 with bad export cookie 10988384969089673213 [ 6665.334274] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6665.338669] LustreError: 335528:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6665.615870] Lustre: server umount lustre-MDT0001 complete [ 6675.460294] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 6677.984938] Key type lgssc unregistered [ 6678.289741] LNet: 338801:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6679.333962] LNet: Removed LNI 192.168.203.134@tcp [ 6680.187924] Key type .llcrypt unregistered [ 6680.190243] Key type ._llcrypt unregistered [ 6703.535085] Key type ._llcrypt registered [ 6703.536844] Key type .llcrypt registered [ 6703.620976] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 6704.459701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6704.484256] alg: No test for adler32 (adler32-zlib) [ 6705.490780] Lustre: Lustre: Build Version: 2.16.57_2_ge787a7d [ 6705.695572] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 6705.700078] LNet: Accept secure, port 988 [ 6707.359193] Key type lgssc registered [ 6708.310311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6715.526437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6715.542878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.977232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6720.109550] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6726.125376] LustreError: 340338:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.147642] LustreError: 340338:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6727.402786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6727.873772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.021836] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6736.547211] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6740.120472] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6746.792646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6747.229959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.223178] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6759.073778] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6772.196536] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.209169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6772.216929] Lustre: Skipped 1 previous similar message [ 6775.378903] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 6776.519578] LustreError: 342614:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6776.610201] Lustre: server umount lustre-MDT0001 complete [ 6777.315176] LustreError: 340334:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.326132] LustreError: 340334:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6779.835369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6780.474352] LustreError: 340335:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.584337] LustreError: 341901:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.602336] LustreError: 341901:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6787.281044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6787.345707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6787.350526] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6787.380666] Lustre: 340328:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6787.594177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.743916] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 6792.708513] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6793.216147] LustreError: 343491:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6793.220266] LustreError: 343491:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6793.230422] LustreError: 340330:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6793.281990] LustreError: 343497:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -17 [ 6793.295206] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6793.295206] [ 7099.878135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7099.878683] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7099.882141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7099.891315] Lustre: Skipped 2 previous similar messages [ 7099.898059] Lustre: Skipped 1 previous similar message [ 7102.031767] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 7107.126749] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 7107.138294] Lustre: Skipped 2 previous similar messages [ 7112.159183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7112.252135] LustreError: 351828:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7112.255870] LustreError: 351828:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7112.345654] Lustre: server umount lustre-MDT0001 complete [ 7115.245740] LustreError: 340333:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.261249] LustreError: 340333:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7115.605968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.636383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7122.766200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7122.768624] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7122.883588] Lustre: 340328:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7123.340679] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7127.704993] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7128.553481] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7132.143235] LustreError: 352793:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7132.168970] LustreError: 352793:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7132.197047] LustreError: 340330:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7132.219652] LustreError: 340330:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7132.347473] LustreError: 352796:0:(obd_config.c:685:class_setup()) Device 8 already setup (type osp) [ 7132.363897] LustreError: 352796:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -17 [ 7132.371571] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.134@tcp [ 7132.371571] [ 7434.858267] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) [ 7434.861446] Lustre: Skipped 3 previous similar messages [ 7435.744977] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.751449] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7440.302908] LustreError: 360896:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7440.305459] LustreError: 360896:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7440.381897] Lustre: server umount lustre-MDT0001 complete [ 7440.867584] LustreError: 345763:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.898308] LustreError: 345763:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 7442.839710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7450.124799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7450.206052] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7450.208477] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7450.254727] Lustre: 340328:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7450.595779] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7454.179628] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7455.720510] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7457.290053] LustreError: 361855:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7457.304363] LustreError: 361855:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7457.325133] LustreError: 340330:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7457.342244] LustreError: 340330:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7457.416457] LustreError: 361858:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7457.439994] LustreError: 361858:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -114 [ 7457.452357] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7457.452357] [ 7778.274927] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.276308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7778.278757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.278765] Lustre: Skipped 3 previous similar messages [ 7778.297222] Lustre: Skipped 2 previous similar messages [ 7780.165193] LustreError: 369723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7780.168291] LustreError: 369723:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7780.269913] Lustre: server umount lustre-MDT0000 complete [ 7783.393156] LustreError: 340333:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7783.416248] LustreError: 340333:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 7783.472438] LustreError: 343180:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752517483 with bad export cookie 15057753652679718113 [ 7783.478262] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7783.481706] LustreError: 343180:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7788.517038] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.532288] Lustre: Skipped 1 previous similar message [ 7789.752469] LustreError: 369925:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7789.763540] LustreError: 369925:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7790.031592] Lustre: server umount lustre-MDT0001 complete [ 7793.313653] Lustre: server umount lustre-OST0000 complete [ 7801.250622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 7808.756618] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7817.150061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7823.208102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7828.933198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7834.601532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7840.542566] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 14:25:38 (1752517538) [ 7846.627397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7846.692219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.883732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7846.927068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7846.977809] Lustre: lustre-MDT0000: new disk, initializing [ 7847.103641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7847.138401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7850.382147] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7859.451230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7859.507906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7859.586530] Lustre: 372650:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7859.621826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7859.625182] Lustre: Skipped 1 previous similar message [ 7859.676992] Lustre: lustre-MDT0001: new disk, initializing [ 7859.746757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7859.779477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7859.783395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7863.093930] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7869.158574] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7872.102177] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7878.136839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7878.206160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7878.477117] Lustre: lustre-OST0000: new disk, initializing [ 7878.480138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7878.553027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.908063] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 7887.416579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7887.429617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7887.521872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7890.576087] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7898.081311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.096089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7898.100693] Lustre: Skipped 1 previous similar message [ 7898.104192] Lustre: Skipped 3 previous similar messages [ 7899.214834] LustreError: 375071:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7899.218089] LustreError: 375071:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7899.315544] Lustre: server umount lustre-MDT0000 complete [ 7903.039691] LustreError: 372642:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752517603 with bad export cookie 15057753652679720080 [ 7903.042315] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7903.042703] LustreError: 373435:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 7903.081852] LustreError: 372642:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7903.211170] LustreError: 372655:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.225243] LustreError: 372655:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7903.456868] Lustre: server umount lustre-MDT0001 complete [ 7909.111442] Lustre: server umount lustre-OST0000 complete [ 7936.163953] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 7944.141657] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 7954.681589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7961.800822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7967.494271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7973.816231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7975.457061] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7817 sec ========== 14:27:53 (1752517673) [ 7977.214601] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:27:55 (1752517675) === [ 7980.888321] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:27:58 (1752517678) === [ 8018.188421] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 8020.944910] Key type lgssc unregistered [ 8021.271213] LNet: 380563:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8022.315705] LNet: Removed LNI 192.168.203.134@tcp [ 8023.219959] Key type .llcrypt unregistered [ 8023.225244] Key type ._llcrypt unregistered