[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468858948 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003154] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005026] kvm-guest: setup PV IPIs [ 0.007744] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011166] LSM: Security Framework initializing [ 0.013052] Yama: becoming mindful. [ 0.014033] SELinux: Initializing. [ 0.015083] *** VALIDATE selinux *** [ 0.027355] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032771] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035492] *** VALIDATE tmpfs *** [ 0.036445] *** VALIDATE proc *** [ 0.039098] *** VALIDATE cgroup *** [ 0.040008] *** VALIDATE cgroup2 *** [ 0.041349] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044050] Spectre V2 : User space: Vulnerable [ 0.045005] Speculative Store Bypass: Vulnerable [ 0.047000] debug: unmapping init [mem 0xffffffffa4459000-0xffffffffa4460fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050025] ... version: 2 [ 0.051029] ... bit width: 48 [ 0.052007] ... generic registers: 4 [ 0.053007] ... value mask: 0000ffffffffffff [ 0.054007] ... max period: 00007fffffffffff [ 0.055009] ... fixed-purpose events: 3 [ 0.056008] ... event mask: 000000070000000f [ 0.057292] rcu: Hierarchical SRCU implementation. [ 0.059687] smp: Bringing up secondary CPUs ... [ 0.060538] x86: Booting SMP configuration: [ 0.061017] .... node #0, CPUs: #1 #2 #3 [ 0.066543] smp: Brought up 1 node, 4 CPUs [ 0.068021] smpboot: Max logical packages: 1 [ 0.069010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143326] node 0 deferred pages initialised in 71ms [ 0.148149] devtmpfs: initialized [ 0.149339] x86/mm: Memory block size: 128MB [ 0.150000] gcov: version magic: 0x41383552 [ 0.152412] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154341] pinctrl core: initialized pinctrl subsystem [ 0.155235] [ 0.155639] ************************************************************* [ 0.156009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ** ** [ 0.158014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159010] ** ** [ 0.160017] ** This means that this kernel is built to expose internal ** [ 0.161012] ** IOMMU data structures, which may compromise security on ** [ 0.162006] ** your system. ** [ 0.163006] ** ** [ 0.164008] ** If you see this message and you are not debugging the ** [ 0.165122] ** kernel, report this immediately to your vendor! ** [ 0.166018] ** ** [ 0.167012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168010] ************************************************************* [ 0.169949] NET: Registered protocol family 16 [ 0.170484] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174574] cpuidle: using governor menu [ 0.176451] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179658] PCI: Using configuration type 1 for base access [ 0.182149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204496] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.209094] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210000] cryptd: max_cpu_qlen set to 1000 [ 0.213285] ACPI: Added _OSI(Module Device) [ 0.215016] ACPI: Added _OSI(Processor Device) [ 0.216016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220545] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227507] ACPI: Interpreter enabled [ 0.228067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230011] ACPI: Using IOAPIC for interrupt routing [ 0.232137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236624] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250544] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265808] acpiphp: Slot [2] registered [ 0.267181] acpiphp: Slot [3] registered [ 0.269087] acpiphp: Slot [4] registered [ 0.270076] acpiphp: Slot [5] registered [ 0.272128] acpiphp: Slot [6] registered [ 0.273083] acpiphp: Slot [7] registered [ 0.274085] acpiphp: Slot [8] registered [ 0.275081] acpiphp: Slot [9] registered [ 0.277100] acpiphp: Slot [10] registered [ 0.278182] acpiphp: Slot [11] registered [ 0.279000] acpiphp: Slot [12] registered [ 0.279000] acpiphp: Slot [13] registered [ 0.281068] acpiphp: Slot [14] registered [ 0.282058] acpiphp: Slot [15] registered [ 0.283060] acpiphp: Slot [16] registered [ 0.285144] acpiphp: Slot [17] registered [ 0.287060] acpiphp: Slot [18] registered [ 0.288061] acpiphp: Slot [19] registered [ 0.289056] acpiphp: Slot [20] registered [ 0.290054] acpiphp: Slot [21] registered [ 0.292062] acpiphp: Slot [22] registered [ 0.293093] acpiphp: Slot [23] registered [ 0.295063] acpiphp: Slot [24] registered [ 0.296061] acpiphp: Slot [25] registered [ 0.297066] acpiphp: Slot [26] registered [ 0.299066] acpiphp: Slot [27] registered [ 0.300064] acpiphp: Slot [28] registered [ 0.301065] acpiphp: Slot [29] registered [ 0.302095] acpiphp: Slot [30] registered [ 0.304069] acpiphp: Slot [31] registered [ 0.305067] PCI host bridge to bus 0000:00 [ 0.306017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315013] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.317019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324900] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329330] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336779] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.350421] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353713] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.355033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358268] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373716] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.377011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.423028] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.438000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.464016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.486020] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.496161] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.611012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.625017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.637017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.646193] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.659023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.675020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.707013] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.720228] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.736012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.746012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.769012] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.779737] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.799024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.811013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.826015] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.837198] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.840383] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.842274] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.845405] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.847215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.850328] iommu: Default domain type: Passthrough [ 0.852852] SCSI subsystem initialized [ 0.854207] ACPI: bus type USB registered [ 0.856067] usbcore: registered new interface driver usbfs [ 0.858036] usbcore: registered new interface driver hub [ 0.860057] usbcore: registered new device driver usb [ 0.862103] pps_core: LinuxPPS API ver. 1 registered [ 0.863006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.867096] PTP clock support registered [ 0.870055] EDAC MC: Ver: 3.0.0 [ 0.871815] PCI: Using ACPI for IRQ routing [ 0.873931] NetLabel: Initializing [ 0.876008] NetLabel: domain hash size = 128 [ 0.878006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.880067] NetLabel: unlabeled traffic allowed by default [ 0.882237] vgaarb: loaded [ 0.883583] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.885007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.895799] clocksource: Switched to clocksource kvm-clock [ 1.069189] VFS: Disk quotas dquot_6.6.0 [ 1.072174] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.074322] *** VALIDATE ramfs *** [ 1.075347] *** VALIDATE hugetlbfs *** [ 1.076947] pnp: PnP ACPI init [ 1.079216] pnp: PnP ACPI: found 6 devices [ 1.115135] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.125149] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.127325] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.129389] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.131502] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.133642] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.136016] NET: Registered protocol family 2 [ 1.138083] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.142398] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.145382] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.150304] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.153264] TCP: Hash tables configured (established 65536 bind 65536) [ 1.155774] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.158430] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.160940] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.163636] NET: Registered protocol family 1 [ 1.168644] RPC: Registered named UNIX socket transport module. [ 1.170528] RPC: Registered udp transport module. [ 1.172322] RPC: Registered tcp transport module. [ 1.173915] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.176827] NET: Registered protocol family 44 [ 1.178345] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.180498] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.182343] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.184502] PCI: CLS 0 bytes, default 64 [ 1.185858] Unpacking initramfs... [ 2.670320] debug: unmapping init [mem 0xffff8f2ffcc54000-0xffff8f2ffffbffff] [ 2.674403] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.676482] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.679253] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.192988] Initialise system trusted keyrings [ 3.194205] Key type blacklist registered [ 3.196617] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.205022] zbud: loaded [ 3.208670] *** VALIDATE nfs *** [ 3.209800] *** VALIDATE nfs4 *** [ 3.211282] pstore: using deflate compression [ 3.214832] Platform Keyring initialized [ 3.355153] NET: Registered protocol family 38 [ 3.360450] Key type asymmetric registered [ 3.361451] Asymmetric key parser 'x509' registered [ 3.362647] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.372690] io scheduler mq-deadline registered [ 3.374743] io scheduler kyber registered [ 3.376784] io scheduler bfq registered [ 3.378727] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.387744] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.395058] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.403566] ACPI: Power Button [PWRF] [ 3.547288] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.642698] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.876275] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.014712] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.293310] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.331324] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.387459] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.396594] Non-volatile memory driver v1.3 [ 4.399730] Linux agpgart interface v0.103 [ 4.447910] virtio_blk virtio1: [vda] 132808 512-byte logical blocks (68.0 MB/64.8 MiB) [ 4.451570] vda: detected capacity change from 0 to 67997696 [ 4.469670] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.475769] vdb: detected capacity change from 0 to 1073741824 [ 4.492520] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.499755] vdc: detected capacity change from 0 to 2621440000 [ 4.516198] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.519413] vdd: detected capacity change from 0 to 2621440000 [ 4.544239] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.548414] vde: detected capacity change from 0 to 4294967296 [ 4.569441] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.572559] vdf: detected capacity change from 0 to 4294967296 [ 4.582107] libphy: Fixed MDIO Bus: probed [ 4.588355] usbcore: registered new interface driver usbserial_generic [ 4.590928] usbserial: USB Serial support registered for generic [ 4.597355] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.601610] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.603510] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.609216] mousedev: PS/2 mouse device common for all mice [ 4.611661] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.613279] rtc_cmos 00:05: RTC can wake from S4 [ 4.623374] rtc_cmos 00:05: registered as rtc0 [ 4.624419] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.628169] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.632272] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.646175] intel_pstate: CPU model not supported [ 4.651156] hid: raw HID events driver (C) Jiri Kosina [ 4.653196] usbcore: registered new interface driver usbhid [ 4.659084] usbhid: USB HID core driver [ 4.660812] drop_monitor: Initializing network drop monitor service [ 4.662625] Initializing XFRM netlink socket [ 4.664221] NET: Registered protocol family 10 [ 4.667340] Segment Routing with IPv6 [ 4.668479] NET: Registered protocol family 17 [ 4.676799] mpls_gso: MPLS GSO support [ 4.684225] RAS: Correctable Errors collector initialized. [ 4.686047] AVX version of gcm_enc/dec engaged. [ 4.688725] AES CTR mode by8 optimization enabled [ 4.822216] sched_clock: Marking stable (4822188977, 0)->(5900678488, -1078489511) [ 4.828142] registered taskstats version 1 [ 4.832300] Loading compiled-in X.509 certificates [ 4.834025] zswap: loaded using pool lzo/zbud [ 4.873855] Key type big_key registered [ 4.890200] Key type encrypted registered [ 4.891501] ima: No TPM chip found, activating TPM-bypass! [ 4.893591] ima: Allocated hash algorithm: sha1 [ 4.895194] ima: No architecture policies found [ 4.896729] evm: Initialising EVM extended attributes: [ 4.900430] evm: security.selinux [ 4.905472] evm: security.ima [ 4.906385] evm: security.capability [ 4.907417] evm: HMAC attrs: 0x1 [ 4.910588] rtc_cmos 00:05: setting system clock to 2025-08-19 20:54:31 UTC (1755636871) [ 4.920424] debug: unmapping init [mem 0xffffffffa5403000-0xffffffffa55fffff] [ 4.922945] debug: unmapping init [mem 0xffffffffa4182000-0xffffffffa4458fff] [ 4.931064] Write protecting the kernel read-only data: 28672k [ 4.936522] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 4.938807] debug: unmapping init [mem 0xffffffffa3114000-0xffffffffa31fffff] [ 4.987566] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.997659] systemd[1]: Detected virtualization kvm. [ 4.999610] systemd[1]: Detected architecture x86-64. [ 5.001679] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.033052] systemd[1]: No hostname configured. [ 5.034828] systemd[1]: Set hostname to . [ 5.037120] random: systemd: uninitialized urandom read (16 bytes read) [ 5.039219] systemd[1]: Initializing machine ID from random generator. [ 5.123360] random: ln: uninitialized urandom read (6 bytes read) [ 5.309210] random: systemd: uninitialized urandom read (16 bytes read) [ 5.312129] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.323212] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 5.330629] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.168506] device-mapper: uevent: version 1.0.3 [ 6.170568] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 7.106843] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.186884] virtio_net virtio0 ens2: renamed from eth0 [ 7.486688] scsi host0: ata_piix [ 7.575532] scsi host1: ata_piix [ 7.584752] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.591135] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.417266] random: crng init done [ 12.418841] random: 7 urandom warning(s) missed due to ratelimiting [ 12.559925] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.945300] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.588742] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.038381] SELinux: Disabled at runtime. [ 16.116315] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.131816] systemd[1]: Detected virtualization kvm. [ 16.138438] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.355927] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.359728] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.372411] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.375918] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.379112] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.400424] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.408451] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ 17.600394] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.368111] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.224313] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.305217] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.568933] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.633592] EDAC sbridge: Ver: 1.1.2 [ 23.462097] Key type dns_resolver registered [ 23.807773] NFS: Registering the id_resolver key type [ 23.809423] Key type id_resolver registered [ 23.810404] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…-only root support (9s / no limit) [ *] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 48.989189] libcfs: loading out-of-tree module taints kernel. [ 48.996088] Key type ._llcrypt registered [ 48.996930] Key type .llcrypt registered [ 49.028695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 55.653365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 56.256088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.263115] alg: No test for adler32 (adler32-zlib) [ 57.235770] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 57.516372] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 59.135111] Key type lgssc registered [ 59.670822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.891212] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.791839] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.136871] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.633627] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.241383] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.274368] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 84.619273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.641567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.652423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.751115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.764340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.806505] Lustre: lustre-MDT0000: new disk, initializing [ 85.831761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.838702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 87.189743] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 92.399815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.432284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 92.463925] Lustre: 6472:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 92.479313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 92.481602] Lustre: Skipped 1 previous similar message [ 92.520237] Lustre: lustre-MDT0001: new disk, initializing [ 92.540509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 92.549245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 92.554357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 93.901948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.203291] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 99.300269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.327740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.409578] Lustre: lustre-OST0000: new disk, initializing [ 99.411388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 99.430379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 101.316653] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.942600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 104.945546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.967183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 106.818815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.842743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.875616] Lustre: lustre-OST0001: new disk, initializing [ 106.878267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 106.896530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.764971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.088511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 111.093649] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 111.108838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 114.644641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.143902] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 126.919482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 128.269833] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 129.625239] Lustre: DEBUG MARKER: Client: 2.16.57.62 [ 130.410269] Lustre: DEBUG MARKER: MDS: 2.16.57.62 [ 131.196388] Lustre: DEBUG MARKER: OSS: 2.16.57.62 [ 131.708969] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 19 16:56:37 EDT 2025 [ 137.204259] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 137.672828] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 152.032088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.032516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 152.037686] Lustre: Skipped 2 previous similar messages [ 152.039927] Lustre: Skipped 1 previous similar message [ 155.451638] LustreError: 12136:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 155.480236] Lustre: server umount lustre-MDT0000 complete [ 156.756644] LustreError: 6464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637023 with bad export cookie 17822399694082077734 [ 156.757852] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 156.760545] LustreError: 6464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 157.151606] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.151992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 157.155902] Lustre: Skipped 2 previous similar messages [ 157.158587] Lustre: Skipped 2 previous similar messages [ 161.183598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 161.185972] Lustre: Skipped 1 previous similar message [ 162.871099] LustreError: 12338:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 162.873511] LustreError: 12338:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 162.937458] Lustre: server umount lustre-MDT0001 complete [ 164.229051] LustreError: 12539:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 164.231537] LustreError: 12539:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 164.250080] Lustre: server umount lustre-OST0000 complete [ 165.558323] Lustre: server umount lustre-OST0001 complete [ 167.559461] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 170.364361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 173.725180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.020896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.016777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.191482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.509748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.528104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.603969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.614255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.645717] Lustre: lustre-MDT0000: new disk, initializing [ 182.670272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.678421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.805083] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 187.790680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.812176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.840345] Lustre: 14905:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.933831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.149406] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 192.219238] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 193.450664] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 195.694216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.720359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.811955] Lustre: lustre-OST0000: new disk, initializing [ 195.813767] Lustre: Skipped 1 previous similar message [ 195.815854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.819060] Lustre: Skipped 2 previous similar messages [ 195.845472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.847201] Lustre: Skipped 1 previous similar message [ 197.367908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.370698] Lustre: Skipped 1 previous similar message [ 197.372448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.409263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.647120] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 200.776552] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 202.175095] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 202.273437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 203.599111] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 203.691777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 206.381100] LustreError: 17810:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 206.383186] LustreError: 17810:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 206.447898] Lustre: server umount lustre-OST0000 complete [ 207.841543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.842369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.845117] Lustre: Skipped 1 previous similar message [ 212.960890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 212.963048] Lustre: Skipped 2 previous similar messages [ 218.084193] LustreError: 18010:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 218.085828] LustreError: 18010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 218.111528] Lustre: server umount lustre-MDT0000 complete [ 219.344298] LustreError: 18012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637085 with bad export cookie 17822399694082079253 [ 219.346949] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 219.350759] LustreError: 18012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 228.835397] LustreError: 18214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 228.837034] LustreError: 18214:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 228.922102] Lustre: server umount lustre-MDT0001 complete [ 230.656539] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 16:58:16 (1755637096) [ 232.975046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.131298] LustreError: 18740:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 233.136761] LustreError: 18740:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 233.153819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.533282] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 237.432645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 238.924426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 240.692405] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 241.884920] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 244.148891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 246.320894] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 248.205802] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 260.667122] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 268.255970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 268.259382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.266248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.268150] Lustre: Skipped 2 previous similar messages [ 269.886948] LustreError: 21042:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 269.888758] LustreError: 21042:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 269.917088] Lustre: server umount lustre-MDT0000 complete [ 271.223537] LustreError: 20345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637137 with bad export cookie 17822399694082080135 [ 271.228030] LustreError: 20345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.228047] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.375444] Lustre: server umount lustre-MDT0001 complete [ 283.059322] Lustre: server umount lustre-OST0000 complete [ 289.767633] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 293.265665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 293.407641] LustreError: 22763:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.423943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 293.426684] Lustre: Skipped 2 previous similar messages [ 294.718275] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 297.691265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 299.164926] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 300.142759] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 302.328560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 304.347142] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 307.239886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 307.267651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 307.318617] Lustre: lustre-OST0001: new disk, initializing [ 307.320514] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 308.929616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 308.932927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 308.968607] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 309.243843] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 313.644591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 321.166656] Lustre: Setting parameter general.obdfilter.*.client_cache_count=256 in log params [ 321.169934] Lustre: Skipped 1 previous similar message [ 324.580888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.582864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.586431] Lustre: Skipped 4 previous similar messages [ 324.589140] Lustre: Skipped 5 previous similar messages [ 330.174074] LustreError: 26220:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 330.176502] LustreError: 26220:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 330.212297] Lustre: server umount lustre-MDT0000 complete [ 331.388066] LustreError: 25019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637197 with bad export cookie 17822399694082080982 [ 331.391835] LustreError: 25019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 331.392086] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.542819] Lustre: server umount lustre-MDT0001 complete [ 343.042592] Lustre: server umount lustre-OST0000 complete [ 353.544055] Lustre: server umount lustre-OST0001 complete [ 359.132106] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 362.440086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.576231] LustreError: 27992:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.598784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.601181] Lustre: Skipped 3 previous similar messages [ 363.831867] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 366.560457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 367.977211] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 368.970534] Lustre: 29096:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 371.193838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 373.270448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 373.412477] LustreError: 29452:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.201858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 378.256377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 381.363434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 382.455233] Lustre: 30940:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 392.160036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 392.163132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.166524] Lustre: Skipped 1 previous similar message [ 392.168317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 392.171366] Lustre: Skipped 4 previous similar messages [ 395.136630] LustreError: 31142:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 395.138979] LustreError: 31142:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 395.175835] Lustre: server umount lustre-MDT0000 complete [ 396.525166] LustreError: 28720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637263 with bad export cookie 17822399694082082277 [ 396.530639] LustreError: 28720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 396.530664] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.670777] Lustre: server umount lustre-MDT0001 complete [ 408.193935] Lustre: server umount lustre-OST0000 complete [ 419.778801] Lustre: server umount lustre-OST0001 complete [ 422.630441] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 17:01:28 (1755637288) [ 431.899333] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 435.162929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.303706] LustreError: 33729:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.675538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 439.519121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 441.010971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 442.033577] Lustre: 34833:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 444.321259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 445.475184] LustreError: 35187:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.377477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 449.324532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 451.470681] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 454.723783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 456.001839] Lustre: 36674:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 465.375910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.378134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.381466] Lustre: Skipped 3 previous similar messages [ 465.384117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.385980] Lustre: Skipped 3 previous similar messages [ 468.449708] Lustre: server umount lustre-MDT0000 complete [ 469.701320] LustreError: 33710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637336 with bad export cookie 17822399694082083383 [ 469.704881] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.707083] LustreError: 33710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 469.850173] Lustre: server umount lustre-MDT0001 complete [ 481.282062] Lustre: server umount lustre-OST0000 complete [ 492.844680] Lustre: server umount lustre-OST0001 complete [ 495.655616] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 17:02:41 (1755637361) [ 500.932821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 504.361457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.496424] LustreError: 38789:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.514898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.517154] Lustre: Skipped 7 previous similar messages [ 505.774524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 508.540885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 509.973927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 510.943897] Lustre: 39893:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 513.126822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 515.106258] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 518.027455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 520.051091] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 523.238826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 529.806184] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 534.502421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.505989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.507396] Lustre: Skipped 5 previous similar messages [ 536.959604] LustreError: 41940:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 536.961570] LustreError: 41940:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 536.985969] Lustre: server umount lustre-MDT0000 complete [ 538.211606] LustreError: 38768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637404 with bad export cookie 17822399694082084489 [ 538.215342] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.215350] LustreError: 38768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.616283] LustreError: 38784:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.623612] LustreError: 38784:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 544.408121] Lustre: server umount lustre-MDT0001 complete [ 547.280453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.743719] LustreError: 3621:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f307ec4b480 x1840918743517440/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 549.230469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 552.166092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 553.670097] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 555.526643] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.787342] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 572.905405] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 572.906833] Lustre: Skipped 1 previous similar message [ 573.699978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 17 sec [ 574.924346] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 575.477206] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 578.019320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 583.001082] Lustre: server umount lustre-MDT0000 complete [ 584.274410] LustreError: 38770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637450 with bad export cookie 17822399694082085630 [ 584.278850] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.424197] Lustre: server umount lustre-MDT0001 complete [ 595.456959] Lustre: server umount lustre-OST0000 complete [ 607.036250] Lustre: server umount lustre-OST0001 complete [ 609.937277] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 17:04:36 (1755637476) [ 615.034529] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 618.652868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.801741] LustreError: 46104:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.808213] LustreError: 46104:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [ 620.078063] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 622.905831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 624.441055] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 625.417542] Lustre: 47211:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 625.420432] Lustre: 47211:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 627.628034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 629.642739] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 632.558659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 634.559425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 637.607761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 644.070993] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 648.671939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 648.674126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.677273] Lustre: Skipped 6 previous similar messages [ 648.679318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.680937] Lustre: Skipped 18 previous similar messages [ 654.672612] Lustre: server umount lustre-MDT0000 complete [ 655.841569] LustreError: 49057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637522 with bad export cookie 17822399694082086435 [ 655.844273] LustreError: 49057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 655.846479] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.006887] Lustre: server umount lustre-MDT0001 complete [ 666.625133] Lustre: server umount lustre-OST0000 complete [ 678.141501] Lustre: server umount lustre-OST0001 complete [ 681.218294] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 17:05:47 (1755637547) [ 681.994272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 682.576088] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 17:05:48 (1755637548) [ 685.899738] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 688.593548] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 691.938692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 693.866760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 696.041272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 696.064169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.144408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 696.182174] Lustre: lustre-MDT0000: new disk, initializing [ 696.209261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 697.409019] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 700.415472] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 702.621195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 702.649211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 704.447142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 704.455362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 704.610351] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 707.695995] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 708.364903] Lustre: DEBUG MARKER: create test files [ 719.841273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 724.511967] Lustre: server umount lustre-OST0000 complete [ 725.881750] Lustre: server umount lustre-MDT0000 complete [ 729.445707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 730.614591] Key type lgssc unregistered [ 730.756777] LNet: 54555:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.758975] LNetError: 54555:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.771457] LNet: Removed LNI 192.168.203.155@tcp [ 731.123106] Key type .llcrypt unregistered [ 731.124155] Key type ._llcrypt unregistered [ 741.025308] Key type ._llcrypt registered [ 741.026223] Key type .llcrypt registered [ 741.064747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 741.435167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.511925] alg: No test for adler32 (adler32-zlib) [ 742.380628] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 742.477221] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 744.063240] Key type lgssc registered [ 744.438705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 747.028507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 747.038436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.204159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.598794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 751.406903] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 753.618940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 753.746768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.662750] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 757.389647] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 757.862958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 757.886880] Lustre: 57121:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 757.889277] Lustre: 57121:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 757.891022] Lustre: 57121:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 757.892773] Lustre: 57121:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 757.894642] Lustre: 57121:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 757.896338] Lustre: 57121:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 765.413827] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 767.013174] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 773.601668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.602499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.608294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.895133] LustreError: 57859:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 774.958030] Lustre: server umount lustre-OST0000 complete [ 776.290373] LustreError: 58061:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 776.292365] LustreError: 58061:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 776.370369] Lustre: server umount lustre-MDT0000 complete [ 780.270749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 781.325594] Key type lgssc unregistered [ 781.454466] LNet: 58587:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.458536] LNetError: 58587:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.468328] LNet: Removed LNI 192.168.203.155@tcp [ 781.787835] Key type .llcrypt unregistered [ 781.789265] Key type ._llcrypt unregistered [ 792.963348] Key type ._llcrypt registered [ 792.964337] Key type .llcrypt registered [ 793.002189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 793.300783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 793.375663] alg: No test for adler32 (adler32-zlib) [ 794.251947] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 794.343168] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 795.919158] Key type lgssc registered [ 796.246874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 798.784073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 798.792634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.940660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 801.225911] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 802.976598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 805.212921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 805.328591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.152147] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 808.898345] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 809.446073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 809.467025] Lustre: 61253:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 809.470089] Lustre: 61253:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 809.472484] Lustre: 61253:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 809.474974] Lustre: 61253:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 809.477869] Lustre: 61253:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 809.481118] Lustre: 61253:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 817.161203] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 819.680315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.680689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.687695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.486680] LustreError: 61942:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 821.543322] Lustre: server umount lustre-OST0000 complete [ 824.901219] LustreError: 62144:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 824.902993] LustreError: 62144:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 825.007420] Lustre: server umount lustre-MDT0000 complete [ 828.565503] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 829.641245] Key type lgssc unregistered [ 829.768327] LNet: 62671:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.772143] LNetError: 62671:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.780292] LNet: Removed LNI 192.168.203.155@tcp [ 830.072767] Key type .llcrypt unregistered [ 830.074035] Key type ._llcrypt unregistered [ 835.844456] Key type ._llcrypt registered [ 835.845828] Key type .llcrypt registered [ 835.883090] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 842.036151] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 842.455545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.518333] alg: No test for adler32 (adler32-zlib) [ 843.390504] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 843.486065] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 845.079156] Key type lgssc registered [ 845.422756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.438249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.715773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.783466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.896975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.133490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.153215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.158298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.250258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 858.261944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 858.307422] Lustre: lustre-MDT0000: new disk, initializing [ 858.331159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.339850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 859.575693] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 863.743809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 863.770702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 863.805094] Lustre: 65758:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 863.831811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 863.834227] Lustre: Skipped 1 previous similar message [ 863.871227] Lustre: lustre-MDT0001: new disk, initializing [ 863.906152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.917296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 863.920751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 865.175133] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 868.278286] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 869.489758] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 871.731048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 871.755136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 871.877212] Lustre: lustre-OST0000: new disk, initializing [ 871.879563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 871.912783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.745331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 875.582605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 875.585979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 875.623340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 876.860582] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 878.220351] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 878.298377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 879.578624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 879.649062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 880.611355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.611567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.614160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.616646] Lustre: Skipped 1 previous similar message [ 880.620019] Lustre: Skipped 1 previous similar message [ 882.350024] LustreError: 68666:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 882.417903] Lustre: server umount lustre-OST0000 complete [ 884.704238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 884.706416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.711556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.728971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.729517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.788575] LustreError: 68866:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 889.791180] LustreError: 68866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 889.826555] Lustre: server umount lustre-MDT0000 complete [ 891.071526] LustreError: 67491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637757 with bad export cookie 17458597141787580348 [ 891.075838] LustreError: 67491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 891.076740] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 891.141515] LustreError: 69068:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 891.143320] LustreError: 69068:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 891.240425] Lustre: server umount lustre-MDT0001 complete [ 894.017117] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 17:09:20 (1755637760) [ 897.197684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 901.476416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 909.424731] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 912.579515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 915.690403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 917.917337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 919.896190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 922.019394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 924.382856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 924.412286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.522399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 924.533539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 924.568960] Lustre: lustre-MDT0000: new disk, initializing [ 924.594798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.603915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 925.907959] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 929.977993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 930.002881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 930.032491] Lustre: 73283:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 930.060252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 930.062726] Lustre: Skipped 1 previous similar message [ 930.097154] Lustre: lustre-MDT0001: new disk, initializing [ 930.131337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 930.142505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 930.145720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 931.445883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 934.638068] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 935.846411] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 938.103850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 938.133656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 938.235475] Lustre: lustre-OST0000: new disk, initializing [ 938.237646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 938.266594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.047568] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 943.054922] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 944.120074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 944.123183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 944.155495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 944.383840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 944.456511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 945.718080] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 945.803625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 949.217700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.219773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.221291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.222930] Lustre: Skipped 1 previous similar message [ 949.225013] Lustre: Skipped 1 previous similar message [ 952.556105] LustreError: 76187:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 952.558233] LustreError: 76187:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 952.632916] Lustre: server umount lustre-OST0000 complete [ 954.336907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.337623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.341922] Lustre: Skipped 1 previous similar message [ 954.343634] Lustre: Skipped 1 previous similar message [ 964.576257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.579795] Lustre: Skipped 4 previous similar messages [ 968.159095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 968.193120] LustreError: 76389:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 968.196333] LustreError: 76389:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 968.226298] Lustre: server umount lustre-MDT0000 complete [ 968.672634] LustreError: 73293:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.676469] LustreError: 73293:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 969.597607] LustreError: 73275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637836 with bad export cookie 17458597141787581216 [ 969.601579] LustreError: 73275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 969.603185] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.768278] Lustre: server umount lustre-MDT0001 complete [ 972.699686] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 17:10:38 (1755637838) [ 975.010744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.170505] LustreError: 77262:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.197434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.674518] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 979.602303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 980.932166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 982.657877] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 983.916082] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 986.162351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 988.183346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 990.000391] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 991.253549] Lustre: 79578:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting c7631630-7561-4834-bceb-95e468a7f9e7 at adminstrative request [ 991.711839] LustreError: MGC192.168.203.155@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 991.713811] LustreError: Skipped 1 previous similar message [ 991.715054] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.718287] Lustre: 77253:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 991.718443] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1001.952949] Lustre: MGS: Client c7631630-7561-4834-bceb-95e468a7f9e7 (at 0@lo) reconnecting [ 1001.957792] Lustre: Evicted from MGS (at 192.168.203.155@tcp) after server handle changed from 0xf2496599d0e61e84 to 0xf2496599d0e620fa [ 1001.962085] Lustre: MGC192.168.203.155@tcp: Connection restored to (at 0@lo) [ 1023.594555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1023.616569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1023.666852] Lustre: lustre-OST0001: new disk, initializing [ 1023.668723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1023.697964] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.699800] Lustre: Skipped 2 previous similar messages [ 1025.419125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1028.375125] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1029.257841] LustreError: 80700:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1029.260061] LustreError: 80700:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1029.275736] Lustre: server umount lustre-OST0001 complete [ 1029.626065] LustreError: 78848:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.623802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1030.625974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.629750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1034.725693] LustreError: 78847:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.733461] LustreError: 78847:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1036.645510] Lustre: server umount lustre-OST0000 complete [ 1052.127089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1052.157595] LustreError: 81113:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 1052.159545] LustreError: 81113:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1052.185691] Lustre: server umount lustre-MDT0000 complete [ 1053.406269] LustreError: 80701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637919 with bad export cookie 17458597141787582714 [ 1053.408916] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.410462] LustreError: 80701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1053.558379] Lustre: server umount lustre-MDT0001 complete [ 1056.412403] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 17:12:02 (1755637922) [ 1056.999241] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1057.602751] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 17:12:03 (1755637923) [ 1058.101643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1058.708414] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 17:12:04 (1755637924) [ 1059.308265] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1059.924337] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 17:12:06 (1755637926) [ 1060.441041] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1062.175361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.546043] loop: module loaded [ 1062.551398] loop0: detected capacity change from 0 to 4294967296 [ 1062.573579] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1063.115602] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1064.160538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1065.506631] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1068.180690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1069.517063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1070.790850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1072.064291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.328213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.355583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1074.367850] LustreError: 83877:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1074.370100] LustreError: 83877:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1074.372521] LustreError: 83877:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1074.374826] LustreError: 83877:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1080.581913] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1083.503922] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1087.095414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1089.336817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1091.420347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1093.565672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1096.023836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1096.045562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.130160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1096.141482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1096.178744] Lustre: lustre-MDT0000: new disk, initializing [ 1096.201214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.208357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1097.551549] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1101.987540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1102.026710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1102.076730] Lustre: 86702:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1102.184459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1103.434114] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1106.532532] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1107.647893] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1109.815031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1109.836395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1110.905961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1110.946513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1111.818849] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1114.968633] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1116.393867] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1116.476516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1117.698195] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1117.790375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1121.252773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.252940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.254805] LustreError: Skipped 2 previous similar messages [ 1121.258590] Lustre: Skipped 3 previous similar messages [ 1121.260346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.262193] Lustre: Skipped 10 previous similar messages [ 1124.524095] LustreError: 89613:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1124.525959] LustreError: 89613:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1124.583019] Lustre: server umount lustre-OST0000 complete [ 1140.191111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1140.242955] Lustre: server umount lustre-MDT0000 complete [ 1140.705153] LustreError: 86713:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.710140] LustreError: 86713:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1141.384116] LustreError: 86694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638007 with bad export cookie 17458597141787583309 [ 1141.386497] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1141.387155] LustreError: 86694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1141.508816] Lustre: server umount lustre-MDT0001 complete [ 1143.990785] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 17:13:30 (1755638010) [ 1146.125312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.483874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1150.113719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1151.481482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1153.167470] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.318455] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.333258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1158.058693] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1159.681846] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1161.033177] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1161.105481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1162.287744] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1162.351683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1164.396632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1164.422899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1164.483127] Lustre: lustre-OST0001: new disk, initializing [ 1164.484689] Lustre: Skipped 2 previous similar messages [ 1164.486573] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1164.487974] Lustre: Skipped 3 previous similar messages [ 1166.284301] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1169.252644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1169.899497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1169.901783] Lustre: Skipped 2 previous similar messages [ 1169.903163] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1169.915047] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1170.441103] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1170.503429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1171.662342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1171.727807] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1181.217120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1186.324587] Lustre: Failing over lustre-MDT0000 [ 1186.622879] Lustre: server umount lustre-MDT0000 complete [ 1188.590728] LustreError: 90682:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.595215] LustreError: 90682:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1190.370729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.373937] Lustre: Skipped 4 previous similar messages [ 1201.128632] LDISKFS-fs (dm-0): recovery complete [ 1201.129967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.165347] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.271760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1202.494429] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1203.940324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1203.940774] Lustre: 92469:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1206.756387] Lustre: 91393:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1206.757689] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 1206.760724] Lustre: 91393:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 1206.783094] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1207.095113] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1207.409452] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1207.413202] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1207.727141] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1207.729175] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 1208.048834] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1208.051763] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 8 previous similar messages [ 1208.680830] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1208.683206] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 1208.999149] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1209.001467] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 1210.255761] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1210.259250] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 6 previous similar messages [ 1210.887705] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1210.890757] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1211.207142] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1211.209860] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 1214.337234] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1214.340770] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 12 previous similar messages [ 1214.968882] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1214.971850] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1215.287092] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1215.289385] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 1222.473515] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1222.477410] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 25 previous similar messages [ 1223.104917] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1223.107519] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1223.423127] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1223.425389] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 1238.730136] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1238.733931] Lustre: 96277:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 51 previous similar messages [ 1239.361408] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1239.364374] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 51 previous similar messages [ 1239.679089] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1239.681599] LustreError: 96277:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 51 previous similar messages [ 1263.095066] Lustre: 96277:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1263.097375] Lustre: 96277:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1263.099576] Lustre: 96277:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 15df8ae3-8d39-4cb3-b3ed-799887f2252b@192.168.203.55@tcp [ 1263.102609] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1263.104387] Lustre: 96277:0:(ldlm_lib.c:1831:abort_req_replay_queue()) @@@ aborted: req@00000000b200139c x1840919561177088/t0(8589934777) o101->15df8ae3-8d39-4cb3-b3ed-799887f2252b@192.168.203.55@tcp:400/0 lens 592/0 e 0 to 0 dl 1755638140 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1263.111184] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1263.114613] Lustre: lustre-MDT0000: Denying connection for new client 15df8ae3-8d39-4cb3-b3ed-799887f2252b (at 192.168.203.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1263.118075] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1263.136660] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1263.136842] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1263.139782] Lustre: Skipped 2 previous similar messages [ 1263.156116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 1263.156155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 1268.193298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.196419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.198754] Lustre: Skipped 1 previous similar message [ 1268.202182] Lustre: Skipped 7 previous similar messages [ 1270.125311] LustreError: 96932:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1270.127009] LustreError: 96932:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1270.142150] Lustre: server umount lustre-OST0000 complete [ 1272.287758] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.314336] LustreError: 92270:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.320021] LustreError: 92270:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 17 previous similar messages [ 1277.409527] Lustre: server umount lustre-OST0001 complete [ 1284.636429] Lustre: server umount lustre-MDT0000 complete [ 1285.788291] LustreError: 90667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638152 with bad export cookie 17458597141787634493 [ 1285.791505] LustreError: 90667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1285.792891] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.935405] Lustre: server umount lustre-MDT0001 complete [ 1288.601449] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 17:15:54 (1755638154) [ 1290.986774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1290.989286] Lustre: *** cfs_fail_loc=197, val=0*** [ 1308.447468] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1314.591342] LustreError: 98226:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 1320.741615] Lustre: *** cfs_fail_loc=197, val=0*** [ 1320.742669] Lustre: Skipped 65 previous similar messages [ 1344.287231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.289045] Lustre: Skipped 7 previous similar messages [ 1346.037761] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1347.701728] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1348.422206] Lustre: server umount lustre-OST0000 complete [ 1351.118149] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 17:16:57 (1755638217) [ 1364.576929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.722897] LustreError: 99728:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.967809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1369.017673] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1371.004902] LustreError: 99759:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1371.009671] LustreError: 99759:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1371.014768] LustreError: 99759:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1371.171174] Lustre: server umount lustre-MDT0000 complete [ 1376.224764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1381.601274] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1384.707654] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1388.133533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.393220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1392.276265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1394.352424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1396.639232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1396.661840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.734595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1396.744789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1396.779123] Lustre: lustre-MDT0000: new disk, initializing [ 1396.812150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1398.159993] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1402.205496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1402.232038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1402.259764] Lustre: 103774:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1402.363432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1403.569680] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1406.508740] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1407.540186] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1409.615066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1409.646842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1411.542271] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1414.516882] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1415.718496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1415.807807] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1415.878964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1417.057276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1417.129022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1420.770442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.772602] LustreError: Skipped 1 previous similar message [ 1420.773678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.774543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.777133] Lustre: Skipped 5 previous similar messages [ 1420.780539] Lustre: Skipped 6 previous similar messages [ 1423.788099] LustreError: 106678:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1423.789787] LustreError: 106678:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 1423.856023] Lustre: server umount lustre-OST0000 complete [ 1431.149574] Lustre: server umount lustre-MDT0000 complete [ 1433.009907] LustreError: 103764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638299 with bad export cookie 17458597141787645119 [ 1433.016474] LustreError: 103764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1433.016810] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.183561] Lustre: server umount lustre-MDT0001 complete [ 1436.741315] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 17:18:22 (1755638302) [ 1440.291540] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1441.800136] Key type lgssc unregistered [ 1441.974758] LNet: 107802:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.979228] LNetError: 107802:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.988823] LNet: Removed LNI 192.168.203.155@tcp [ 1442.477317] Key type .llcrypt unregistered [ 1442.479775] Key type ._llcrypt unregistered [ 1449.753764] Key type ._llcrypt registered [ 1449.755470] Key type .llcrypt registered [ 1449.805344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1456.167743] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1456.557872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.564825] alg: No test for adler32 (adler32-zlib) [ 1457.430812] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1457.517096] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1459.103137] Key type lgssc registered [ 1459.443363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1462.548394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1464.941901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1467.608982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1469.878655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.279674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1474.757959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1474.779035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1474.783536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.885923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1475.900242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1475.946257] Lustre: lustre-MDT0000: new disk, initializing [ 1475.975282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.984670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1477.387075] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1481.357422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1481.382923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1481.474816] Lustre: lustre-OST0000: new disk, initializing [ 1481.478193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1481.480940] Lustre: Skipped 1 previous similar message [ 1481.504555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.303810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1484.029172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1484.031680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1484.041386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1487.367634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1487.393425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1487.442820] Lustre: lustre-OST0001: new disk, initializing [ 1487.445649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1487.471327] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1489.293458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1490.242456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1490.244927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1490.252563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1495.521496] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.522510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.524535] Lustre: Skipped 1 previous similar message [ 1495.527090] Lustre: Skipped 1 previous similar message [ 1499.263147] LustreError: 113324:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1499.347548] Lustre: server umount lustre-MDT0000 complete [ 1501.388756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1502.889134] LustreError: 111100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638369 with bad export cookie 7893441365388064521 [ 1502.891080] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1502.892748] LustreError: 111100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1502.915089] LustreError: 113919:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1502.916797] LustreError: 113919:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1502.930404] Lustre: server umount lustre-OST0000 complete [ 1504.073050] LustreError: 114119:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1504.074986] LustreError: 114119:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1504.137287] Lustre: server umount lustre-OST0001 complete [ 1506.870109] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 17:19:33 (1755638373) [ 1509.709936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1512.609319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1512.959514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1514.802853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1515.175601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1517.318505] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 17:19:43 (1755638383) [ 1521.250620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1521.842263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1522.450270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.038374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.326126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.902577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.059203] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1532.438842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1535.680892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1537.860303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1539.858056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1541.929515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1544.170892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1544.198313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.273971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1544.284311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1544.317406] Lustre: lustre-MDT0000: new disk, initializing [ 1544.340645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.346643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1545.603610] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1549.511261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1549.534155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1549.561471] Lustre: 119413:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1549.579995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1549.581495] Lustre: Skipped 1 previous similar message [ 1549.614507] Lustre: lustre-MDT0001: new disk, initializing [ 1549.645985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.659608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1549.662409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1550.796722] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1553.761596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1554.958915] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1557.154340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1557.178301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1558.516943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1558.520121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1558.554102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1559.021788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1561.978025] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1563.330270] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1563.405397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1564.682895] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1564.763903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1567.405699] LustreError: 122320:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1567.407465] LustreError: 122320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1567.468495] Lustre: server umount lustre-OST0000 complete [ 1568.736344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.736804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.739730] Lustre: Skipped 1 previous similar message [ 1573.856448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1573.858324] Lustre: Skipped 2 previous similar messages [ 1578.976446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1578.978255] Lustre: Skipped 1 previous similar message [ 1578.979991] LustreError: 122521:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1578.982056] LustreError: 122521:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1579.005881] Lustre: server umount lustre-MDT0000 complete [ 1580.238990] LustreError: 119403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638446 with bad export cookie 7893441365388065732 [ 1580.245435] LustreError: 119403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1580.246232] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.730372] LustreError: 122724:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1589.732411] LustreError: 122724:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1589.806784] Lustre: server umount lustre-MDT0001 complete [ 1592.925417] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 17:20:59 (1755638459) [ 1595.399246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.610284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.612456] Lustre: Skipped 1 previous similar message [ 1596.906780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1599.543380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1600.821918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1602.421996] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.718123] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1605.951170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1607.760583] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1609.459321] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1615.547592] Lustre: *** cfs_fail_loc=159, val=0*** [ 1615.548935] Lustre: Skipped 2 previous similar messages [ 1631.983927] Lustre: lustre-MDT0000: Client 77018cd6-ab16-4830-92ab-80a5c167b872 (at 192.168.203.55@tcp) reconnecting [ 1632.525113] Lustre: *** cfs_fail_loc=159, val=0*** [ 1632.527258] Lustre: Skipped 3 previous similar messages [ 1648.893939] Lustre: lustre-MDT0000: Client 77018cd6-ab16-4830-92ab-80a5c167b872 (at 192.168.203.55@tcp) reconnecting [ 1666.531495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.531939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.535433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.538103] Lustre: Skipped 1 previous similar message [ 1669.804491] LustreError: 125869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1669.807805] LustreError: 125869:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1669.868163] Lustre: server umount lustre-OST0000 complete [ 1671.648776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.648783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.653939] Lustre: Skipped 2 previous similar messages [ 1676.768041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.769971] Lustre: Skipped 3 previous similar messages [ 1677.080599] Lustre: server umount lustre-MDT0000 complete [ 1678.274968] LustreError: 125002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638544 with bad export cookie 7893441365388066600 [ 1678.278175] LustreError: 125002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1678.278649] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1678.423292] Lustre: server umount lustre-MDT0001 complete [ 1682.252556] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1683.426204] Key type lgssc unregistered [ 1683.575367] LNet: 126799:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.577516] LNetError: 126799:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.589426] LNet: Removed LNI 192.168.203.155@tcp [ 1683.926191] Key type .llcrypt unregistered [ 1683.927820] Key type ._llcrypt unregistered [ 1692.177778] Key type ._llcrypt registered [ 1692.179380] Key type .llcrypt registered [ 1692.212452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1692.596156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1692.605189] alg: No test for adler32 (adler32-zlib) [ 1693.450242] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1693.534226] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1695.111099] Key type lgssc registered [ 1695.474629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1697.861201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.866801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.012834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.261567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1702.899414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1703.020791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1704.217926] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1705.873450] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1707.041879] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1709.270576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1709.377190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.438896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1711.101612] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1712.801173] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1718.329772] Lustre: *** cfs_fail_loc=159, val=0*** [ 1734.893087] Lustre: lustre-MDT0000: Client 81df1234-89b1-45a6-8aa8-258085ac0cad (at 192.168.203.55@tcp) reconnecting [ 1735.554209] Lustre: *** cfs_fail_loc=159, val=0*** [ 1751.795403] Lustre: lustre-MDT0001: Client 81df1234-89b1-45a6-8aa8-258085ac0cad (at 192.168.203.55@tcp) reconnecting [ 1752.428256] Lustre: *** cfs_fail_loc=159, val=0*** [ 1752.430128] Lustre: Skipped 4 previous similar messages [ 1768.687754] Lustre: lustre-MDT0001: Client 81df1234-89b1-45a6-8aa8-258085ac0cad (at 192.168.203.55@tcp) reconnecting [ 1769.822030] Lustre: 130847:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1770.590696] Lustre: *** cfs_fail_loc=159, val=0*** [ 1770.592046] Lustre: Skipped 3 previous similar messages [ 1787.121791] Lustre: lustre-MDT0000: Client fabc6944-c529-4844-89d7-24ffd3f44dbd (at 192.168.203.55@tcp) reconnecting [ 1787.760517] Lustre: *** cfs_fail_loc=159, val=0*** [ 1787.762120] Lustre: Skipped 14 previous similar messages [ 1804.016639] Lustre: lustre-MDT0000: Client fabc6944-c529-4844-89d7-24ffd3f44dbd (at 192.168.203.55@tcp) reconnecting [ 1804.531357] Lustre: 131146:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1807.840059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1807.842331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.848717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.890203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.892860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.893079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.899684] Lustre: Skipped 1 previous similar message [ 1811.822255] LustreError: 131245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1811.886318] Lustre: server umount lustre-OST0000 complete [ 1815.010031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.011057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.015494] Lustre: Skipped 1 previous similar message [ 1819.322289] LustreError: 131446:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1819.324012] LustreError: 131446:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1819.346593] Lustre: server umount lustre-MDT0000 complete [ 1820.128954] LustreError: 128166:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1820.133065] LustreError: 128166:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1820.532227] LustreError: 130974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638687 with bad export cookie 12960978260179767592 [ 1820.536085] LustreError: 130974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1820.536618] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.601408] LustreError: 131649:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1820.603231] LustreError: 131649:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1820.680289] Lustre: server umount lustre-MDT0001 complete [ 1824.346267] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1825.360761] Key type lgssc unregistered [ 1825.495362] LNet: 132175:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.497503] LNetError: 132175:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.506269] LNet: Removed LNI 192.168.203.155@tcp [ 1825.813675] Key type .llcrypt unregistered [ 1825.815320] Key type ._llcrypt unregistered [ 1834.463914] Key type ._llcrypt registered [ 1834.464977] Key type .llcrypt registered [ 1834.496971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1834.923706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.935605] alg: No test for adler32 (adler32-zlib) [ 1835.776257] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1835.849508] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1837.423115] Key type lgssc registered [ 1837.748176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.578575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.586717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.741553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.119773] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1846.054310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1846.187775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.485072] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1849.213924] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1850.414827] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1852.590439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1852.697577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.509274] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1855.788927] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 1856.236598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1867.053813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 1874.243105] Lustre: 135795:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1874.833236] Lustre: 135844:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1875.273331] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 1876.433628] Lustre: 135989:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 1876.758665] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 1876.761337] Lustre: Skipped 1 previous similar message [ 1877.076418] Lustre: 136087:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1880.172108] LustreError: 136184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1880.239278] Lustre: server umount lustre-OST0000 complete [ 1882.080191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.084731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.092974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.593963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.603164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.713724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.715969] Lustre: Skipped 1 previous similar message [ 1891.813777] LustreError: 136385:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1891.820588] LustreError: 136385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1891.863154] Lustre: server umount lustre-MDT0000 complete [ 1892.834225] LustreError: 133540:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.842103] LustreError: 133540:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1893.709301] LustreError: 133520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638760 with bad export cookie 13529071235230691242 [ 1893.715432] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1893.715901] LustreError: 133520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.099428] LustreError: 136592:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1904.102057] LustreError: 136592:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1904.188833] Lustre: server umount lustre-MDT0001 complete [ 1907.725986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1908.967441] Key type lgssc unregistered [ 1909.085368] LNet: 137119:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.087764] LNetError: 137119:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.096234] LNet: Removed LNI 192.168.203.155@tcp [ 1909.382414] Key type .llcrypt unregistered [ 1909.383516] Key type ._llcrypt unregistered [ 1919.311625] Key type ._llcrypt registered [ 1919.313282] Key type .llcrypt registered [ 1919.362598] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1919.813715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1919.820533] alg: No test for adler32 (adler32-zlib) [ 1920.677267] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1920.784910] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1922.367125] Key type lgssc registered [ 1922.780822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1926.092369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1926.098064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.247393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.407662] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1931.123753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1931.237438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1932.421842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1934.101569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1935.292125] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1937.425781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1937.539739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.340263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1940.653953] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 1941.993650] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1951.909804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 1955.049910] Lustre: *** cfs_fail_loc=159, val=0*** [ 1970.415313] Lustre: lustre-MDT0000: Client 75cf7ce7-60eb-4276-9376-dc014d718d36 (at 192.168.203.55@tcp) reconnecting [ 1971.679715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1971.681790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.685276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.707068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.707645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.707931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.709925] Lustre: Skipped 1 previous similar message [ 1977.714120] LustreError: 140865:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1977.817164] Lustre: server umount lustre-OST0000 complete [ 1982.431910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.433981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.437693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1985.803985] LustreError: 141067:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1985.807722] LustreError: 141067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1985.844879] Lustre: server umount lustre-MDT0000 complete [ 1987.541833] LustreError: 140097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638854 with bad export cookie 17484942499021589280 [ 1987.548264] LustreError: 140097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1987.548485] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1987.664246] LustreError: 141270:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1987.667439] LustreError: 141270:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1987.775238] Lustre: server umount lustre-MDT0001 complete [ 1992.395442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1993.769849] Key type lgssc unregistered [ 1993.925660] LNet: 141797:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.930569] LNetError: 141797:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.944595] LNet: Removed LNI 192.168.203.155@tcp [ 1994.431994] Key type .llcrypt unregistered [ 1994.434440] Key type ._llcrypt unregistered [ 2004.282692] Key type ._llcrypt registered [ 2004.283678] Key type .llcrypt registered [ 2004.319607] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2004.720217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.725839] alg: No test for adler32 (adler32-zlib) [ 2005.567731] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2005.656168] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2007.239200] Key type lgssc registered [ 2007.580373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.062626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.069206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.216672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.510430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2015.397808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2015.513813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.728374] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2018.556230] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2019.961263] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2023.108486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2023.310712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.759354] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2026.413959] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2028.322239] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2035.625229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2051.794828] Lustre: lustre-OST0000: evicting a6622112-46cd-4c73-b155-d8b5f6c596c7 (at 192.168.203.55@tcp) by administrative request [ 2051.797797] Lustre: Skipped 1 previous similar message [ 2076.642358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2076.645036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.646827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.648889] Lustre: Skipped 1 previous similar message [ 2081.763787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2081.765640] Lustre: Skipped 2 previous similar messages [ 2082.419225] LustreError: 145948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2082.519427] Lustre: server umount lustre-OST0000 complete [ 2086.881948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.882690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.884758] Lustre: Skipped 1 previous similar message [ 2090.367420] LustreError: 146149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2090.369222] LustreError: 146149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2090.398947] Lustre: server umount lustre-MDT0000 complete [ 2091.607656] LustreError: 143144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638958 with bad export cookie 12702989560496181786 [ 2091.611929] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.613073] LustreError: 143144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2091.679683] LustreError: 146352:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2091.682911] LustreError: 146352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2091.767443] Lustre: server umount lustre-MDT0001 complete [ 2095.447303] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2096.401681] Key type lgssc unregistered [ 2096.529274] LNet: 146878:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.531353] LNetError: 146878:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.540280] LNet: Removed LNI 192.168.203.155@tcp [ 2096.803534] Key type .llcrypt unregistered [ 2096.804540] Key type ._llcrypt unregistered [ 2121.895027] Key type ._llcrypt registered [ 2121.896152] Key type .llcrypt registered [ 2121.932826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2122.218357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2122.231222] alg: No test for adler32 (adler32-zlib) [ 2123.101955] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2123.211073] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2124.807171] Key type lgssc registered [ 2125.185994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.457318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.464471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.649579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.078605] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2134.267736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2134.382192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.818781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2137.651483] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2139.126904] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2141.530724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2141.658962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.508411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2143.718955] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2145.451888] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2154.983330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2159.072123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.074328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.078057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2160.101879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2160.103667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2160.104080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.590242] LustreError: 151576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2164.662449] Lustre: server umount lustre-OST0000 complete [ 2170.336367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2170.336781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.337478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.337482] Lustre: Skipped 1 previous similar message [ 2170.344489] Lustre: Skipped 1 previous similar message [ 2171.963176] LustreError: 151778:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2171.965288] LustreError: 151778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2171.990089] Lustre: server umount lustre-MDT0000 complete [ 2173.142207] LustreError: 149299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639039 with bad export cookie 9469932355818867079 [ 2173.145924] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.148211] LustreError: 149299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2173.210850] LustreError: 151980:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2173.212515] LustreError: 151980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2173.298070] Lustre: server umount lustre-MDT0001 complete [ 2176.924319] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2178.047202] Key type lgssc unregistered [ 2178.176331] LNet: 152506:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.178470] LNetError: 152506:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2178.186384] LNet: Removed LNI 192.168.203.155@tcp [ 2178.519924] Key type .llcrypt unregistered [ 2178.521596] Key type ._llcrypt unregistered [ 2186.889293] Key type ._llcrypt registered [ 2186.891079] Key type .llcrypt registered [ 2186.931973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2187.274747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.291928] alg: No test for adler32 (adler32-zlib) [ 2188.168715] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2188.256095] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2189.839163] Key type lgssc registered [ 2190.215551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2192.546914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2194.817464] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 17:31:01 (1755639061) [ 2199.747235] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2202.765779] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2206.122787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2208.256239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2210.223216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2212.336248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2215.018688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2215.049101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2215.056391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.140580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2216.152568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2216.186703] Lustre: lustre-MDT0000: new disk, initializing [ 2216.214414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.222376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2217.457651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2221.327928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2221.348214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2221.375309] Lustre: 156680:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2221.395714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2221.397324] Lustre: Skipped 1 previous similar message [ 2221.440502] Lustre: lustre-MDT0001: new disk, initializing [ 2221.473949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.484445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2221.487249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2222.696201] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2225.599877] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2226.645685] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2228.708169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2228.729272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2228.817784] Lustre: lustre-OST0000: new disk, initializing [ 2228.819504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2228.843758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.530186] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2231.734138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2231.736891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2231.769199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2233.463523] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2236.898907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2236.899028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.900921] LustreError: Skipped 1 previous similar message [ 2236.904595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.906769] Lustre: Skipped 1 previous similar message [ 2242.018084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2242.019755] Lustre: Skipped 1 previous similar message [ 2242.798046] LustreError: 159247:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2242.858969] Lustre: server umount lustre-OST0000 complete [ 2247.136852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2247.137038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.137874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.144117] Lustre: Skipped 2 previous similar messages [ 2250.105598] LustreError: 159450:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2250.107324] LustreError: 159450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2250.132431] Lustre: server umount lustre-MDT0000 complete [ 2251.241961] LustreError: 156671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639117 with bad export cookie 1575760644492322078 [ 2251.245121] LustreError: 156671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2251.245832] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2251.305456] LustreError: 159651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2251.307208] LustreError: 159651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2251.380658] Lustre: server umount lustre-MDT0001 complete [ 2254.812948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2255.879471] Key type lgssc unregistered [ 2256.021465] LNet: 160178:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.023574] LNetError: 160178:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.030374] LNet: Removed LNI 192.168.203.155@tcp [ 2256.330330] Key type .llcrypt unregistered [ 2256.331310] Key type ._llcrypt unregistered [ 2265.064704] Key type ._llcrypt registered [ 2265.066498] Key type .llcrypt registered [ 2265.106101] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2265.521358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2265.550723] alg: No test for adler32 (adler32-zlib) [ 2266.414673] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2266.498395] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2268.071126] Key type lgssc registered [ 2268.432110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2270.960244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2270.966497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.128243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.378799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2276.085941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2276.214677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.406740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2278.992181] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2280.083766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2282.117279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2282.218513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.239153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2283.888120] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2285.463576] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2288.661811] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2288.668273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.671765] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2288.673895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2288.677321] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2288.995635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2289.276387] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2289.278509] Lustre: Skipped 1 previous similar message [ 2289.282982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.286038] Lustre: Skipped 1 previous similar message [ 2289.287313] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2289.289010] Lustre: Skipped 1 previous similar message [ 2289.290162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2289.292950] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2289.295657] Lustre: Skipped 1 previous similar message [ 2289.555783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2289.827963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2290.095748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2290.362559] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2290.364640] Lustre: Skipped 3 previous similar messages [ 2290.368767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.371632] Lustre: Skipped 3 previous similar messages [ 2290.372780] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2290.374584] Lustre: Skipped 3 previous similar messages [ 2290.375708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2290.378659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2290.381064] Lustre: Skipped 3 previous similar messages [ 2290.648158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2290.909776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2291.181622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2291.470027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2291.735061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2291.994259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2292.254169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2292.522350] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2292.524437] Lustre: Skipped 7 previous similar messages [ 2292.529023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.531907] Lustre: Skipped 7 previous similar messages [ 2292.533148] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2292.534790] Lustre: Skipped 7 previous similar messages [ 2292.535985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2292.538874] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2292.540675] Lustre: Skipped 7 previous similar messages [ 2292.804554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2293.070472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2293.340695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2293.630789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2293.929253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2294.242351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2294.561361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2294.866220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2295.129102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2295.398183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2295.657821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2295.919265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2296.186839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2296.459945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2296.723077] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2296.724712] Lustre: Skipped 14 previous similar messages [ 2296.728663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.731509] Lustre: Skipped 14 previous similar messages [ 2296.732851] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2296.734517] Lustre: Skipped 14 previous similar messages [ 2296.735834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2296.739043] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2296.742293] Lustre: Skipped 14 previous similar messages [ 2297.014144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2297.296129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2297.595048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2297.867445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2298.148081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2298.420915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2298.697286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2298.982605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2299.254262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2299.525244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2299.808610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2300.099802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2300.380449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2300.650597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2300.932750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2301.209942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2301.484667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2301.769851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2302.046979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2302.322297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2302.599393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2302.869443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2303.136411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2303.399406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2303.667020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2303.934707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2304.213707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2304.493498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2304.766424] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2304.768695] Lustre: Skipped 28 previous similar messages [ 2304.773143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.776369] Lustre: Skipped 28 previous similar messages [ 2304.777628] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2304.779328] Lustre: Skipped 28 previous similar messages [ 2304.780544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2304.783326] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2304.785120] Lustre: Skipped 28 previous similar messages [ 2305.051669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2305.319089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2305.582094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2305.843458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2306.110828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2306.377892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2306.642993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2306.914071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2307.176516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2307.446380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2307.708970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2307.981460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2308.260628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2308.544316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2308.809042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2309.084201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2309.349395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2309.617664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2309.883244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2310.153319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2310.425790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2310.709508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2310.975578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2311.254381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2311.519492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2311.780316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2312.043792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2312.318176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2312.583559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2312.846164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2313.116515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2313.390252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2313.658132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2313.929572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2314.195247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2314.463231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2314.725854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2314.989779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2315.253888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2315.530942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2315.796022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2316.075849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2316.337295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2316.608851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2316.879972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2317.156866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2317.431606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2317.703267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2317.976343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2318.252695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2318.527994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2318.835882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2319.121070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2319.405189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2319.686422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2319.979263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2320.264784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2320.561126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2320.852187] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2320.853932] Lustre: Skipped 58 previous similar messages [ 2320.858220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.861195] Lustre: Skipped 58 previous similar messages [ 2320.862496] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2320.864216] Lustre: Skipped 58 previous similar messages [ 2320.865541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2320.870850] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2320.874677] Lustre: Skipped 58 previous similar messages [ 2321.177433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2321.482842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2321.788692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2322.077051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2322.366303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2322.643301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2322.922483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2323.202351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2323.489700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2323.764880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2324.041304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2324.324030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2324.623161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2324.901957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2325.175232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2325.455786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2325.743460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2326.016084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2326.288016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2326.568224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2326.844818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2327.149986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2327.484841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2327.775529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2328.073648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2328.357448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2328.630768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2328.922160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2329.222251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2329.496580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2329.772128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2330.050537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2330.345139] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2330.347715] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2330.349296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2330.641421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2330.942262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2331.243589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2331.549413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2331.823621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2332.104376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2332.381044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2332.712199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2333.008777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2333.322603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2333.600161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2333.916206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2334.210140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2334.492873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2334.780865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2335.061604] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2335.064721] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2335.065312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2335.339450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2335.605141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2335.878270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.163606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.432457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.714863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.988130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2337.262942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2337.536878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2337.832897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.130637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.418165] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2338.420779] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2338.421695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.703939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.992519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2339.272995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2339.562616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2339.849738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.138026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.430464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.737241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.024697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.322871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.613161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.931983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2342.233160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2342.528124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2342.853364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.144689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.423646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.712249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.008863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.299045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.585974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.897636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.186709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.481954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.759025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.028184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.307790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.592754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.869820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.153539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.439761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.779081] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2347.783076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.789110] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2348.115289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2348.456529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2348.802190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.091256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.372181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.656218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.931455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.212924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.490726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.770555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.038824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.316812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.590463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.859811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.132732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.406456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.678675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.945936] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2352.947666] Lustre: Skipped 110 previous similar messages [ 2352.952171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.955215] Lustre: Skipped 110 previous similar messages [ 2352.956525] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2352.958242] Lustre: Skipped 110 previous similar messages [ 2352.959486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.962360] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2352.964605] Lustre: Skipped 110 previous similar messages [ 2353.230732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2353.521744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2353.821844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2354.144848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2354.438783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2354.734157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.015328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.290640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.569558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.855892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.124831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.407730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.691376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.991579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2357.284732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2357.589116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2357.894075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.200321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.503123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.832282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2359.154593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2359.486492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2359.783081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.062476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.339248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.623965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.899420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.175175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.450446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.728154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.994298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2362.277485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2362.557843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2362.843022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.114150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.387545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.662688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.946222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2364.209308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2364.476306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2364.749954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.032225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.298271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.597791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.909627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2366.204397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2366.481824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2366.756318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.032879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.304844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.561680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.844147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.113245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.397824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.663076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.938264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.217505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.488498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.770355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.057236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.333881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.639231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.954674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2371.214860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2371.478388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2371.742029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.009179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.278486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.541857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.808108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.070527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.341611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.613877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.875795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.137426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.396322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.657452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.925427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.183310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.438595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.695566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.972575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2376.245473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2376.502767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2376.759303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.019672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.286668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.567565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.847976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.127300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.403781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.675394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.947249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2379.224060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2379.482888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2379.756649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.022573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.302889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.608474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.897082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.175838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.494706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.801105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2382.101823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2382.394231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2382.719926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.030794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.337544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.646124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.965608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.276356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.550867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.829636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.126111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.401248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.672890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.959556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2386.248213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2386.528801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2386.791099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.053506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.316266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.573837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.830276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.098900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.372652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.647931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.916118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.177022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.447147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.712232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.986543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2390.259333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2390.526968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2390.785902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.065403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.330875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.592484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.853113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.110244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.370714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.643747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.911951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.174989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.440349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.701040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.961493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.219870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.479135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.739243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.997528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2395.259498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2395.516033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2395.784325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.043765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.299790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.559672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.815820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.079616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.335954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.596330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.862348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.123310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.384402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.651767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.908899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.169890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.428300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.686960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.943542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2400.205403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2400.486441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2400.751301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.015064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.281383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.545469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.805793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.066254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.324445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.581623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.856625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.119256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.383503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.641184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.900450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.156782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.417471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.676340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.933868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.212024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.470885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.733964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.992682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2406.253549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2406.524345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2406.813183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2407.137170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2407.471790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2407.790504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.116451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.397329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.691893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2409.071339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2409.481379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2409.886351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2410.243804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2410.635699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2411.049073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2411.440565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2411.807280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2412.175036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2412.531172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2412.875768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2413.259979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2413.654778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.019875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.342386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.616495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.974335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2415.309450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2415.580654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2415.859374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2416.148888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2416.441492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2416.727168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.000163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2417.001844] Lustre: Skipped 225 previous similar messages [ 2417.005819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.008883] Lustre: Skipped 225 previous similar messages [ 2417.010269] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2417.012061] Lustre: Skipped 225 previous similar messages [ 2417.013267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.016229] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2417.018084] Lustre: Skipped 225 previous similar messages [ 2417.283502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.556895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.844545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.121884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.384926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.664851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.975986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2419.258417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2419.587742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2419.882794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.201395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.512641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.812361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2421.112601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2421.432364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2421.764376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.079855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.366590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.659410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.981151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.276354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.550341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.825848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.120440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.423630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.736560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.022959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.319984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.614986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.901274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2426.178877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2426.448792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2426.775854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.063527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.359889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.666361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.952500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.222502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.502040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.771783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2429.044335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2429.322419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2429.594428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2429.865633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.147942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.432214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.706801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.984293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2431.262648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2431.540756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2431.825800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.094488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.367173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.646846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.919853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2433.198305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2433.475420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2433.739755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.010546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.283901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.553842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.827697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.102067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.381488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.665866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.988286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2436.270407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2436.559830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2436.842722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2437.142624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2437.456164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2437.735375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.011345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.290099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.600070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.912927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.197781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.495447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.787788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.082617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.405359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.682977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.966094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2441.239912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2441.509802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2441.794340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.094327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.378605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.661750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.964502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2443.298440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2443.605306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2443.914841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.199938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.520267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.811532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.085788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.364059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.661136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.948291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.240309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.543093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.839365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2447.130990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2447.416497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2447.712973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.003183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.287877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.574426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.864770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.151438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.422152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.712854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.015708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.294793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.590342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.882169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2451.182170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2451.497925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2451.777695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.078166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.374102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.656165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.938368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.222052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.521311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.829334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2454.121577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2454.432262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2454.729555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.051851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.352916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.649252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.969379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.288496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.579779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.845089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.108038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.379490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.645305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.916723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2458.198340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2458.481653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2458.758865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.034511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.318194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.608488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.890154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2460.282304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2460.649886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.019396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.409559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.760157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.079751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.350973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.625113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.912172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.189220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.476617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.764036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.072051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.362820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.647774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.935652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2465.223528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2465.526140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2465.818354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.112299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.422170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.715867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.007642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.307451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.598027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.873155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2468.157217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2468.453400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2468.765279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.089298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.401287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.693567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.999994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2470.286670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2470.592608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2470.870525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.169049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.448305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.714394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.983109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.250633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.511929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.782125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.047829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.325424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.592311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.874613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.153266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.442100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.721424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.997065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.276948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.546713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.822548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.089447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.365661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.650276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.922164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.212996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.502465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.800187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.082292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.383068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.672378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.975325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2479.273275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2479.568989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2479.870188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.151117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.415917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.683307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.949590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2481.215548] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2481.218946] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2481.219259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2481.484675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2481.753380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.018591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.281833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.555077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.820468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.086424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.370137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.643441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.908913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2484.177317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2484.451499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2484.731290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.026860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.330291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.613042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.881066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.151893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.415838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.686239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.951358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2487.214907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2487.494848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2487.766899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.037707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.300731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.576838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.879470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.149994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.417683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.692283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.973503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2490.244839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2490.516867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2490.804482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.118879] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2491.121761] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2491.122466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.393801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.661071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.928464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.202670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.468716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.733286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.999725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.270923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.542380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.809236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.072967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.341172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.613811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.895691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.173500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.452862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.732216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.019851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.303302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.593265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.878557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2497.172315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2497.446858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2497.749474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.033671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.317741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.599176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.886330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2499.176797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2499.462524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2499.753304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.050414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.347312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.641127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.928835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.222108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.498727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.769383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.055222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.336849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.630285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.938954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.236506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.529106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.812102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.101735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.391784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.697302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.991665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.280329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.565937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.866884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.150177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.431979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.717065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.005423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.280541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.560764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.842195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.127957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.402205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.677295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.950491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.240936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.519083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.810908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.122387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.442345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.750647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.049522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.352872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.668047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.951659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.229158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.515532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.808369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.094906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.376693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.672278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.958616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.247674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.528351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.811132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.105830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.381352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.651905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.932190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.217219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.495317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.765711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.029785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.302672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.566265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.831719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.095907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.365253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.634467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.905477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.177417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.449047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.721522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.985241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.246459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.545542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.835659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.111336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.386714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.658291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.941107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.219567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.510148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.793455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.075750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.349515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.626391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.908473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.186330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.457167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.741527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.020856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.297816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.567995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.843778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.123585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.406959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.679149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.959323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.246131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.528096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.803963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.080514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.359086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.640891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.920164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.196993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.480034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.760596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.031491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.322268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.600651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.873358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.155482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.441125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.718641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.989552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.259621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.534357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.817139] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2532.819923] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) Skipped 1 previous similar message [ 2532.820960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.822390] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2532.828194] LustreError: 161563:0:(osp_precreate.c:1358:osp_precreate_thread()) Skipped 1 previous similar message [ 2533.100226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.368059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.642361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.913164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.192387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.464421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.736716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.026508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.369444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.667343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.943584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.234407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.513591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.812363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.101618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.399138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.676107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.955747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.227276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.512107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.795964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.069796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.347747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.636077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.917606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.223644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.531225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.829437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.106754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.386808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.676639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.968391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.237309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.506401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.780640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.051034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.320124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.590508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.865315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.132236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.409676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.680862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.950897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.222853] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2545.224783] Lustre: Skipped 448 previous similar messages [ 2545.229411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.232505] Lustre: Skipped 448 previous similar messages [ 2545.233874] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2545.235558] Lustre: Skipped 448 previous similar messages [ 2545.236638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.240141] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.155@tcp (at 0@lo) [ 2545.242331] Lustre: Skipped 448 previous similar messages [ 2545.507599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.787066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.059803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.346483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.630530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.908508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.181346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.454260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.726686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.000829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.263817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.526197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.793514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.061076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.331301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.614965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.888329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.149826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.424489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.713806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.984723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.271516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.561359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.861438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.145786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.424839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.712191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.003864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.289270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.574313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.852665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.144378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.420145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.706232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.984533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.267630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.544950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.836676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.116779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.401142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.671963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.945902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.214612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.493170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.767254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.047567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.326877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.603527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.874666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.159364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.435244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.718597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.996237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.270693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.562763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.862216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.157107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.439848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.726970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.025535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.303247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.583750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.865333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.158210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.438655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.716238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.004506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.291597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.572535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.868057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.154487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.441874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.729069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.012888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.295180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.582963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.870617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.155475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.442536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.736748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.021821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.306206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.592719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.881654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.160364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.437684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.733215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.023887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.302642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.581498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.858948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.156763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.439628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.734823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.039894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.316889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2663.946954] LustreError: 167260:0:(ofd_dev.c:1776:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 2663.950347] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 2663.955605] LustreError: 161563:0:(osp_precreate.c:654:osp_precreate_send()) Skipped 1 previous similar message [ 2730.463728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2730.466299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2731.781085] LustreError: 215247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2731.870783] Lustre: server umount lustre-OST0000 complete [ 2735.108272] LustreError: 215448:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2735.109997] LustreError: 215448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2735.188989] Lustre: server umount lustre-MDT0000 complete [ 2736.285220] LustreError: 215248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639602 with bad export cookie 15028747549888871721 [ 2736.288160] LustreError: 215248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2736.289072] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.826630] LustreError: 215649:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2745.828581] LustreError: 215649:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2745.902195] Lustre: server umount lustre-MDT0001 complete [ 2749.736626] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2751.019418] Key type lgssc unregistered [ 2751.179677] LNet: 216176:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.184392] LNetError: 216176:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.194476] LNet: Removed LNI 192.168.203.155@tcp [ 2751.671928] Key type .llcrypt unregistered [ 2751.673979] Key type ._llcrypt unregistered [ 2761.530974] Key type ._llcrypt registered [ 2761.532012] Key type .llcrypt registered [ 2761.562553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2761.834933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2761.855624] alg: No test for adler32 (adler32-zlib) [ 2762.691660] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2762.762647] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2764.335086] Key type lgssc registered [ 2764.621125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2766.881163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2766.888781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.038250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.108129] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2771.710497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2771.830317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2772.963322] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2774.531918] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2775.552035] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2777.635141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2777.742589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.792282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 2779.929173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2782.293097] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2786.145731] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2786.937173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.941376] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2786.944028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2786.947546] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2786.947606] LustreError: 217560:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2789.611180] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 2792.418954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.419436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2792.419885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.431948] Lustre: Skipped 1 previous similar message [ 2793.581180] LustreError: 219920:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2793.642562] Lustre: server umount lustre-OST0000 complete [ 2807.031056] LustreError: 217560:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2821.429210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2821.534601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.155294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2824.748889] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2830.393744] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 2831.455966] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c9a207e3800.ost_server_uuid 50 [ 2831.903739] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c9a207e3800.ost_server_uuid in FULL state after 0 sec [ 2833.825207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2833.831715] Lustre: Skipped 2 previous similar messages [ 2838.827845] LustreError: 221231:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2838.829597] LustreError: 221231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2838.881458] Lustre: server umount lustre-OST0000 complete [ 2843.615772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.617791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.621097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.248088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.249716] Lustre: Skipped 3 previous similar messages [ 2854.367109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2854.396052] LustreError: 221433:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2854.398781] LustreError: 221433:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2854.427102] Lustre: server umount lustre-MDT0000 complete [ 2855.535087] LustreError: 217520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639722 with bad export cookie 12402289086427299323 [ 2855.538240] LustreError: 217520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2855.538385] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2855.664429] Lustre: server umount lustre-MDT0001 complete [ 2859.059480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2859.982273] Key type lgssc unregistered [ 2860.107307] LNet: 222161:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.109406] LNetError: 222161:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.117287] LNet: Removed LNI 192.168.203.155@tcp [ 2860.394499] Key type .llcrypt unregistered [ 2860.395453] Key type ._llcrypt unregistered [ 2870.503548] Key type ._llcrypt registered [ 2870.504592] Key type .llcrypt registered [ 2870.536979] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2870.803033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2870.811562] alg: No test for adler32 (adler32-zlib) [ 2871.646652] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2871.718781] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2873.295115] Key type lgssc registered [ 2873.615082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2876.387146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2876.392659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.537035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.677211] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2881.314114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2881.420718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2882.542951] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2883.419154] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2885.449050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2885.566439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.272434] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2887.658951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 2890.283820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2890.305525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2890.353315] Lustre: lustre-OST0001: new disk, initializing [ 2890.354929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2890.383697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2891.642322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2891.645354] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2891.674746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2892.094393] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2897.315584] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2900.816064] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2917.348112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.348112] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.348826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2917.349593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.351605] Lustre: Skipped 2 previous similar messages [ 2919.099813] LustreError: 227176:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2919.136095] Lustre: server umount lustre-MDT0000 complete [ 2920.306537] LustreError: 223820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639786 with bad export cookie 5047812339576663780 [ 2920.311237] LustreError: 224555:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 2920.312067] LustreError: 223820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2920.312658] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.383834] LustreError: 227378:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2920.386334] LustreError: 227378:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2920.467636] Lustre: server umount lustre-MDT0001 complete [ 2931.698123] LustreError: 227579:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2931.699838] LustreError: 227579:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2931.716057] Lustre: server umount lustre-OST0000 complete [ 2943.088634] LustreError: 227780:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2943.090954] LustreError: 227780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2943.157444] Lustre: server umount lustre-OST0001 complete [ 2944.125860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2944.161292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.501856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2944.540218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.876924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2944.893663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2945.242626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2945.275812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2949.885630] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2952.827311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.964749] LustreError: 229182:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.970377] LustreError: 229182:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2952.984842] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.112919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2956.766166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2958.032271] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2958.900102] Lustre: 230279:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2960.864221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2962.654352] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2963.043346] LustreError: 230635:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.046402] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49761) [ 2965.255174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2965.320105] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2965.321884] Lustre: Skipped 2 previous similar messages [ 2967.065792] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2968.042313] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 2968.042460] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2973.832183] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2975.004343] Lustre: 232114:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2987.487747] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2987.489809] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.493486] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.495057] Lustre: Skipped 3 previous similar messages [ 2988.515042] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.515679] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2988.520948] Lustre: Skipped 2 previous similar messages [ 2988.522570] Lustre: Skipped 2 previous similar messages [ 2990.336579] LustreError: 232425:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2990.338518] LustreError: 232425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2990.366731] Lustre: server umount mylustre-MDT0000 complete [ 2991.463567] LustreError: 229156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639858 with bad export cookie 5047812339576666594 [ 2991.467202] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2991.605531] Lustre: server umount mylustre-MDT0001 complete [ 3002.808086] LustreError: 232827:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3002.809978] LustreError: 232827:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3002.826373] Lustre: server umount mylustre-OST0000 complete [ 3014.253746] Lustre: server umount mylustre-OST0001 complete [ 3015.207323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3015.238848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3015.568727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3015.599339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3015.941433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3015.972790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3016.287916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3016.309100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.302720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3023.252531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.404707] LustreError: 234430:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.426820] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.542207] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3027.061527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3028.314361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3029.163140] Lustre: 235529:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3031.110604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3032.898133] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3033.255498] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49793) [ 3034.275129] LustreError: 235883:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.482844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3036.587369] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 3036.588339] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 3037.265974] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3040.163946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3041.205731] Lustre: 237362:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3047.904583] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3047.906531] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.909798] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3047.911208] Lustre: Skipped 1 previous similar message [ 3052.000671] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3052.002766] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.005591] Lustre: Skipped 2 previous similar messages [ 3054.077699] LustreError: 237612:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3054.080367] LustreError: 237612:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3054.108642] Lustre: server umount tfs-MDT0000 complete [ 3055.201087] LustreError: 234406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639921 with bad export cookie 5047812339576668778 [ 3055.204226] LustreError: 234406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3055.204546] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.332684] Lustre: server umount tfs-MDT0001 complete [ 3067.325272] Lustre: server umount tfs-OST0000 complete [ 3077.682119] Lustre: server umount tfs-OST0001 complete [ 3078.633510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3078.661947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3078.991320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3079.020655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3079.355947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3079.378457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3079.735661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3079.760278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3084.892924] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3087.849994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.003954] LustreError: 239617:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.025568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.027607] Lustre: Skipped 3 previous similar messages [ 3089.145619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3091.958839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3093.491919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3094.567852] Lustre: 240715:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3097.227770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3098.410659] LustreError: 241219:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.415218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49825) [ 3099.647585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3101.492136] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 3103.202539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3104.499466] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 3105.395696] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3109.006168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3110.391670] Lustre: 242549:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3118.257280] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 17:46:24 (1755639984) [ 3133.407728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3133.410056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.413942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.415663] Lustre: Skipped 3 previous similar messages [ 3136.251259] LustreError: 242917:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3136.253538] LustreError: 242917:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3136.281547] Lustre: server umount lustre-MDT0000 complete [ 3137.403385] LustreError: 240717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640003 with bad export cookie 5047812339576670696 [ 3137.406604] LustreError: 240717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3137.406682] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.039991] LustreError: 241091:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.044442] LustreError: 241091:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3143.614147] Lustre: server umount lustre-MDT0001 complete [ 3150.903665] Lustre: server umount lustre-OST0000 complete [ 3158.117393] Lustre: server umount lustre-OST0001 complete [ 3160.038936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3162.929254] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3166.095464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3168.413232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3170.395203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3172.287066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.272601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.291298] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3174.291298] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3174.291298] [ 3174.297575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3174.368958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3174.378331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3174.411367] Lustre: lustre-MDT0000: new disk, initializing [ 3174.433289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.435260] Lustre: Skipped 3 previous similar messages [ 3174.440194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3175.598707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3179.435765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3179.457994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3179.486616] Lustre: 245691:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3179.507303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3179.509016] Lustre: Skipped 1 previous similar message [ 3179.543465] Lustre: lustre-MDT0001: new disk, initializing [ 3179.584143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3179.587246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3180.720955] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3184.661203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3184.684838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.775518] Lustre: lustre-OST0000: new disk, initializing [ 3184.777865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3185.845868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3185.849172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3185.876538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3186.613161] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3190.865951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3190.891467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3190.941209] Lustre: lustre-OST0001: new disk, initializing [ 3190.942944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3192.702693] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3192.770938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3192.774965] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3192.804178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3197.395559] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 17:47:43 (1755640063) [ 3201.904660] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 17:47:47 (1755640067) [ 3216.864201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3216.864303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.866674] LustreError: Skipped 1 previous similar message [ 3216.870302] Lustre: Skipped 5 previous similar messages [ 3216.872169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3216.874797] Lustre: Skipped 8 previous similar messages [ 3219.755105] LustreError: 249092:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3219.756798] LustreError: 249092:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3219.770646] Lustre: server umount lustre-OST0000 complete [ 3223.522042] LustreError: 247423:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.526814] LustreError: 247423:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3227.285591] Lustre: server umount lustre-MDT0000 complete [ 3228.567121] LustreError: 247233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640095 with bad export cookie 5047812339576672474 [ 3228.569979] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.570779] LustreError: 247233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3234.799662] Lustre: server umount lustre-MDT0001 complete [ 3238.838576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3248.233101] Lustre: server umount lustre-OST0001 complete [ 3250.285015] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3257.146043] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3261.307090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3263.535937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3265.453969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3267.816835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3270.760358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3270.794880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.922090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3270.940616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3270.995038] Lustre: lustre-MDT0000: new disk, initializing [ 3271.027585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3272.813845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3277.551187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3277.582582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3277.623281] Lustre: 252533:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3277.733456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3279.324471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3282.944112] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3284.457629] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3286.735392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3286.758059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3288.182528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3288.186801] Lustre: Skipped 1 previous similar message [ 3288.188983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3288.216063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3288.643110] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3291.643744] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3298.785819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.786446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.788192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.788195] Lustre: Skipped 7 previous similar messages [ 3298.789483] LustreError: Skipped 1 previous similar message [ 3298.792636] Lustre: Skipped 5 previous similar messages [ 3300.257929] Lustre: server umount lustre-OST0000 complete [ 3307.473159] Lustre: server umount lustre-MDT0000 complete [ 3308.718120] LustreError: 252525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640175 with bad export cookie 5047812339576673944 [ 3308.720803] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.724050] LustreError: 252525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3308.862464] Lustre: server umount lustre-MDT0001 complete [ 3312.601473] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3313.601568] Key type lgssc unregistered [ 3313.754423] LNet: 255885:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3313.756975] LNetError: 255885:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3313.767400] LNet: Removed LNI 192.168.203.155@tcp [ 3314.100093] Key type .llcrypt unregistered [ 3314.101261] Key type ._llcrypt unregistered [ 3324.264178] Key type ._llcrypt registered [ 3324.265321] Key type .llcrypt registered [ 3324.296260] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3324.578313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.622065] alg: No test for adler32 (adler32-zlib) [ 3325.500397] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3325.590602] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3327.175089] Key type lgssc registered [ 3327.471582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3329.847467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3329.854640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.013259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.188720] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3335.070077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3335.198050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3336.324428] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3338.312885] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.887690] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3342.280945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3342.394061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.167041] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3345.927671] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.410167] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3349.473135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.474550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3349.478048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.068212] LustreError: 259569:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3353.127016] Lustre: server umount lustre-OST0000 complete [ 3354.592222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.592785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.596173] Lustre: Skipped 1 previous similar message [ 3359.711894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.714657] Lustre: Skipped 2 previous similar messages [ 3360.458744] LustreError: 259778:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3360.464076] LustreError: 259778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3360.499730] Lustre: server umount lustre-MDT0000 complete [ 3361.904322] LustreError: 259471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640228 with bad export cookie 10751068139175292491 [ 3361.906935] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.911055] LustreError: 259471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3361.965534] LustreError: 259981:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3361.967605] LustreError: 259981:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3362.063545] Lustre: server umount lustre-MDT0001 complete [ 3366.438679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3367.627288] Key type lgssc unregistered [ 3367.778326] LNet: 260507:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.780616] LNetError: 260507:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3367.790313] LNet: Removed LNI 192.168.203.155@tcp [ 3368.138155] Key type .llcrypt unregistered [ 3368.139251] Key type ._llcrypt unregistered [ 3375.451379] Key type ._llcrypt registered [ 3375.452493] Key type .llcrypt registered [ 3375.496918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3375.985966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3375.992935] alg: No test for adler32 (adler32-zlib) [ 3376.876222] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3376.975525] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3378.575118] Key type lgssc registered [ 3378.892976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3381.751436] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3381.758223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3398.239223] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3404.383172] LustreError: 261650:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 3434.079126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.672772] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3437.209450] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3437.849135] LustreError: 262308:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3437.904865] Lustre: server umount lustre-OST0000 complete [ 3443.377900] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3444.288150] Key type lgssc unregistered [ 3444.417582] LNet: 263138:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3444.421973] LNetError: 263138:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3444.431523] LNet: Removed LNI 192.168.203.155@tcp [ 3444.779219] Key type .llcrypt unregistered [ 3444.780412] Key type ._llcrypt unregistered [ 3459.440744] Key type ._llcrypt registered [ 3459.441717] Key type .llcrypt registered [ 3459.472119] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3459.865529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3459.874301] alg: No test for adler32 (adler32-zlib) [ 3460.713679] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3460.786276] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3462.359098] Key type lgssc registered [ 3462.661431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3466.813401] loop0: detected capacity change from 0 to 204800000 [ 3466.861594] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3466.865781] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3466.986444] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3466.990250] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.008987] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.025815] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.038431] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.059326] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.082617] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.111767] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.468315] loop0: detected capacity change from 0 to 204800000 [ 3467.518645] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3467.739229] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3468.109096] loop0: detected capacity change from 0 to 204800000 [ 3468.232874] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3468.580539] loop0: detected capacity change from 0 to 204800000 [ 3468.710677] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3469.034126] loop0: detected capacity change from 0 to 204800000 [ 3469.044424] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3470.294805] loop0: detected capacity change from 0 to 204800000 [ 3470.305602] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3471.547498] loop0: detected capacity change from 0 to 204800000 [ 3471.557934] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3472.807223] loop0: detected capacity change from 0 to 204800000 [ 3472.817876] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3474.073872] loop0: detected capacity change from 0 to 204800000 [ 3474.102473] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3474.136417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3474.141696] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.160949] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3474.164358] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3474.167775] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3474.171318] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3474.174926] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3474.178638] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3474.182716] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3474.186301] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3474.189831] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3474.193136] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3474.196394] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3474.199911] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3474.203330] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3474.207161] Lustre: 266507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3476.729917] LustreError: 266773:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3476.765670] Lustre: server umount MGS complete [ 3477.083237] loop0: detected capacity change from 0 to 204800000 [ 3477.120686] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.269169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.283183] Lustre: lustre-MDT0000: Aborting client recovery [ 3477.578849] loop1: detected capacity change from 0 to 204800000 [ 3477.628475] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3477.652773] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3477.673263] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3477.676698] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3477.680229] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3477.683385] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3477.686399] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3477.690174] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3477.693298] Lustre: 266943:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3477.699790] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3477.788803] Lustre: lustre-MDT0001: Aborting client recovery [ 3478.103138] loop2: detected capacity change from 0 to 204800000 [ 3478.166765] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.182077] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3478.197670] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3478.200878] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3478.203841] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3478.207579] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3478.210937] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3478.214449] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3478.217679] Lustre: 267029:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3478.265844] Lustre: Found index 0 for lustre-OST0000, updating log [ 3478.292594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.294509] Lustre: Skipped 1 previous similar message [ 3478.299665] Lustre: lustre-OST0000: Aborting recovery [ 3478.602041] loop3: detected capacity change from 0 to 204800000 [ 3478.638037] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.672258] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3478.688927] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3478.692221] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3478.695501] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3478.699291] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3478.702817] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3478.706494] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3478.710065] Lustre: 267117:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3484.135104] LustreError: 266886:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 3487.202339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3487.202716] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3487.218309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3487.219955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3488.226177] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3488.226177] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3489.491462] LustreError: 266854:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 221us: rc = -2 [ 3503.584523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.585128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3503.585342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.587947] Lustre: Skipped 1 previous similar message [ 3506.809577] LustreError: 267474:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3506.811430] LustreError: 267474:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3506.845722] Lustre: server umount lustre-MDT0000 complete [ 3507.126535] LustreError: 266837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640373 with bad export cookie 6153235649001098060 [ 3507.127993] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3507.129760] LustreError: 266837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3508.703572] LustreError: 266850:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3508.703871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.704197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3508.704202] Lustre: Skipped 3 previous similar messages [ 3508.710212] LustreError: 266850:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3508.715088] Lustre: Skipped 1 previous similar message [ 3512.735513] LustreError: 266850:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.735680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3512.742411] LustreError: 266850:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3512.745132] Lustre: Skipped 2 previous similar messages [ 3513.272333] LustreError: 267528:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3513.274467] LustreError: 267528:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3513.358730] Lustre: server umount lustre-MDT0001 complete [ 3519.723073] LustreError: 267580:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3519.725165] LustreError: 267580:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3519.748750] Lustre: server umount lustre-OST0000 complete [ 3534.303068] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3534.322213] LustreError: 267632:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3534.323711] LustreError: 267632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3534.387513] Lustre: server umount lustre-OST0001 complete [ 3536.990365] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 17:53:23 (1755640403) [ 3546.152511] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3549.866679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3553.450321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.924236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3558.178675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3560.122953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.515661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.538211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.614663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3562.625292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3562.660273] Lustre: lustre-MDT0000: new disk, initializing [ 3562.685153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.687212] Lustre: Skipped 1 previous similar message [ 3562.692771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3563.913695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3568.161528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.185764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3568.215830] Lustre: 271174:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3568.237320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3568.239061] Lustre: Skipped 1 previous similar message [ 3568.274168] Lustre: lustre-MDT0001: new disk, initializing [ 3568.304622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3568.316455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3568.319529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3569.559185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3572.909082] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3574.402413] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3576.919920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3576.947065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3577.036888] Lustre: lustre-OST0000: new disk, initializing [ 3577.039091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3577.065885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.102019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3578.105208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3578.136670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3578.888852] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3582.280312] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3585.031600] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3589.076880] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3595.300345] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3601.952517] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 3639.778586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.778679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3639.779251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3639.784491] Lustre: Skipped 1 previous similar message [ 3639.786689] LustreError: Skipped 1 previous similar message [ 3642.094114] LustreError: 274671:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3642.096472] LustreError: 274671:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3642.157412] Lustre: server umount lustre-OST0000 complete [ 3644.898167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.899584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3644.907723] Lustre: Skipped 1 previous similar message [ 3644.913105] Lustre: Skipped 2 previous similar messages [ 3649.708988] Lustre: server umount lustre-MDT0000 complete [ 3650.016927] LustreError: 271184:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.022593] LustreError: 271184:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3651.124072] LustreError: 271166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640517 with bad export cookie 6153235649001100769 [ 3651.126970] LustreError: 271166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.127117] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.261113] Lustre: server umount lustre-MDT0001 complete [ 3654.830943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.413153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3658.559479] LustreError: 276009:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3658.563145] LustreError: 276009:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3658.583595] Lustre: server umount MGS complete [ 3663.368349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.561896] LustreError: 276632:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.570400] LustreError: 276644:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3663.598228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.081327] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3668.427873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3669.916943] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3672.265084] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3673.848181] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3676.620737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3678.779921] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3680.918879] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3710.951005] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.951592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.952769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.962282] Lustre: Skipped 1 previous similar message [ 3716.530180] LustreError: 279291:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3716.534382] LustreError: 279291:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3716.634237] Lustre: server umount lustre-OST0000 complete [ 3719.648523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3719.653704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3724.655518] Lustre: server umount lustre-MDT0000 complete [ 3726.305770] LustreError: 277338:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.315745] LustreError: 277338:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3726.458128] LustreError: 276612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640593 with bad export cookie 6153235649001102596 [ 3726.462588] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.464806] LustreError: 276612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3726.656594] Lustre: server umount lustre-MDT0001 complete [ 3730.977537] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3732.588480] Key type lgssc unregistered [ 3732.774604] LNet: 280220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3732.779469] LNetError: 280220:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3732.788625] LNet: Removed LNI 192.168.203.155@tcp [ 3733.302161] Key type .llcrypt unregistered [ 3733.304050] Key type ._llcrypt unregistered [ 3749.535369] Key type ._llcrypt registered [ 3749.537266] Key type .llcrypt registered [ 3749.590742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3758.184608] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3758.513305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3758.600689] alg: No test for adler32 (adler32-zlib) [ 3759.463925] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3759.562272] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3761.135212] Key type lgssc registered [ 3761.663907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3766.359418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3769.883840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3773.053364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3775.942079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3779.153976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3779.181073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3779.190398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.297868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3780.313905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3780.366386] Lustre: lustre-MDT0000: new disk, initializing [ 3780.402318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.410933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3782.059845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3787.300333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3787.348079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3787.399054] Lustre: 284113:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3787.437880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3787.441485] Lustre: Skipped 1 previous similar message [ 3787.485280] Lustre: lustre-MDT0001: new disk, initializing [ 3787.535489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3787.558350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3787.564183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3789.133319] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3792.739149] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3794.477097] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3797.259236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3797.300202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3797.456601] Lustre: lustre-OST0000: new disk, initializing [ 3797.459729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3797.512503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.995901] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3802.639726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3802.648002] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3802.703682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3804.047190] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3808.103906] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3811.237476] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3817.755199] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3858.917586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3858.918152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.922052] LustreError: Skipped 1 previous similar message [ 3858.923252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3858.929936] Lustre: Skipped 1 previous similar message [ 3863.535242] LustreError: 287560:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3863.617569] Lustre: server umount lustre-OST0000 complete [ 3869.153726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.154968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.163879] Lustre: Skipped 1 previous similar message [ 3869.168582] Lustre: Skipped 2 previous similar messages [ 3871.501270] LustreError: 287761:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3871.505098] LustreError: 287761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3871.544553] Lustre: server umount lustre-MDT0000 complete [ 3873.359881] LustreError: 284103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640739 with bad export cookie 9134849586816031576 [ 3873.364793] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3873.366399] LustreError: 284103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3873.467351] LustreError: 287963:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3873.471051] LustreError: 287963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3873.591525] Lustre: server umount lustre-MDT0001 complete [ 3877.509582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.337893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3881.441948] LustreError: 288900:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3881.445751] LustreError: 288900:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3881.479291] Lustre: server umount MGS complete [ 3886.947505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.183811] LustreError: 289534:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3887.224724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.887534] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3892.621503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3892.703708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3892.745503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3894.094738] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3896.103734] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3897.485945] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3899.935198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3902.170159] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3904.354428] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3934.175889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3934.179562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.186041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3938.277487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.278273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3938.278405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.889129] LustreError: 292181:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3939.893386] LustreError: 292181:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3939.971724] Lustre: server umount lustre-OST0000 complete [ 3943.394271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.400700] Lustre: Skipped 1 previous similar message [ 3948.513101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.517118] Lustre: Skipped 4 previous similar messages [ 3956.191105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3956.226263] LustreError: 292382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3956.229784] LustreError: 292382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3956.255599] Lustre: server umount lustre-MDT0000 complete [ 3957.475158] LustreError: 291130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640824 with bad export cookie 9134849586816033403 [ 3957.477676] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.478848] LustreError: 291130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3957.626243] Lustre: server umount lustre-MDT0001 complete [ 3961.330924] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3962.239666] Key type lgssc unregistered [ 3962.366340] LNet: 293112:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.368787] LNetError: 293112:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3962.382271] LNet: Removed LNI 192.168.203.155@tcp [ 3962.692611] Key type .llcrypt unregistered [ 3962.693731] Key type ._llcrypt unregistered [ 3969.691041] Key type ._llcrypt registered [ 3969.692427] Key type .llcrypt registered [ 3969.731264] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3977.219167] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3977.616635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3977.629741] alg: No test for adler32 (adler32-zlib) [ 3978.524981] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3978.615172] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3980.199179] Key type lgssc registered [ 3980.627223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.925509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.568515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.423276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3993.727888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3996.045111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3996.076472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3996.081380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.184826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3997.201083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3997.239197] Lustre: lustre-MDT0000: new disk, initializing [ 3997.272664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.279989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3998.687194] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4003.721964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4003.758406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4003.916882] Lustre: lustre-OST0000: new disk, initializing [ 4003.920304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4003.923408] Lustre: Skipped 1 previous similar message [ 4003.962867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.775989] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4013.077825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4013.084709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4013.097971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4287.962406] LustreError: 298020:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4288.206665] Lustre: server umount lustre-MDT0000 complete [ 4290.113136] LustreError: 296268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641156 with bad export cookie 18053537126560467940 [ 4290.120439] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4300.284512] LustreError: 298224:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4300.289526] LustreError: 298224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4300.424827] Lustre: server umount lustre-OST0000 complete [ 4312.261766] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4316.814690] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4321.309666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4324.788765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4327.652132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4330.893072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4334.273781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4334.317883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.460155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4334.481294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4334.534084] Lustre: lustre-MDT0000: new disk, initializing [ 4334.575146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.587101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4336.430074] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4342.132970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.156281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4342.187301] Lustre: 301251:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4342.208113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4342.209722] Lustre: Skipped 1 previous similar message [ 4342.248979] Lustre: lustre-MDT0001: new disk, initializing [ 4342.288510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4342.299722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4342.302880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4343.906968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4347.738624] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4349.411346] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4352.644628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4352.691570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4352.820699] Lustre: lustre-OST0000: new disk, initializing [ 4352.823515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4352.865210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.308246] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4359.190892] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4360.203276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4360.209333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4360.270485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4361.003986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4361.135762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4363.119614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4363.260890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4365.282366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4365.286110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.292200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4365.296286] Lustre: Skipped 1 previous similar message [ 4366.259130] LustreError: 304158:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4366.263087] LustreError: 304158:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4366.352899] Lustre: server umount lustre-OST0000 complete [ 4370.403960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.405144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.410826] Lustre: Skipped 1 previous similar message [ 4370.414959] Lustre: Skipped 1 previous similar message [ 4374.348467] LustreError: 304359:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4374.353036] LustreError: 304359:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4374.399283] Lustre: server umount lustre-MDT0000 complete [ 4376.254759] LustreError: 301243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641242 with bad export cookie 18053537126563408731 [ 4376.264597] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.459714] Lustre: server umount lustre-MDT0001 complete [ 4378.161360] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 4378.885903] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 18:07:24 (1755641244) [ 4382.056537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.307245] LustreError: 305132:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.317890] LustreError: 305132:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4382.352593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.365684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4387.807830] LustreError: 305132:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4388.610878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4390.836998] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4393.446691] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4395.338262] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4398.722820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4398.917440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4398.920359] Lustre: Skipped 1 previous similar message [ 4401.510974] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4404.023311] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4407.398530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4407.443739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4407.532629] Lustre: lustre-OST0001: new disk, initializing [ 4407.536182] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4409.179543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4409.186484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4409.215300] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4410.154740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4410.276274] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4410.280338] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 4411.301951] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4411.307920] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4413.350547] LustreError: 307807:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4413.356759] LustreError: 307807:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4414.839488] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4414.932266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4416.849277] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c9a07017800.ost_server_uuid 50 [ 4417.444303] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4417.448804] LustreError: 307803:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 4417.635277] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c9a07017800.ost_server_uuid in FULL state after 0 sec [ 4419.583644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c9a07017800.ost_server_uuid 50 [ 4420.441494] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c9a07017800.ost_server_uuid in FULL state after 0 sec [ 4421.555766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4433.650567] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 4434.919415] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.761120] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 4438.766895] Lustre: Skipped 2 previous similar messages [ 4443.882164] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 4443.886584] Lustre: Skipped 2 previous similar messages [ 4447.199106] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4447.231113] LustreError: 308912:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4447.234778] LustreError: 308912:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4447.266448] Lustre: server umount lustre-OST0001 complete [ 4450.274878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.281104] LustreError: 308466:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.283460] Lustre: Skipped 2 previous similar messages [ 4450.292296] LustreError: 308466:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4455.397360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4455.398771] LustreError: 307808:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.401342] Lustre: Skipped 5 previous similar messages [ 4455.410064] LustreError: 307808:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4455.599143] LustreError: 309115:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4455.602854] LustreError: 309115:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4455.710674] Lustre: server umount lustre-OST0000 complete [ 4460.515266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.515515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4460.527065] LustreError: Skipped 1 previous similar message [ 4463.736056] Lustre: server umount lustre-MDT0000 complete [ 4465.543883] LustreError: 305113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641332 with bad export cookie 18053537126563409599 [ 4465.548856] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.550714] LustreError: 305113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4465.765405] Lustre: server umount lustre-MDT0001 complete [ 4470.535935] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4472.067364] Key type lgssc unregistered [ 4472.234570] LNet: 310044:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4472.239890] LNetError: 310044:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4472.250574] LNet: Removed LNI 192.168.203.155@tcp [ 4472.750188] Key type .llcrypt unregistered [ 4472.752892] Key type ._llcrypt unregistered [ 4484.127818] Key type ._llcrypt registered [ 4484.129688] Key type .llcrypt registered [ 4484.181718] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4484.475761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4484.516730] alg: No test for adler32 (adler32-zlib) [ 4485.380921] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4485.486502] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4487.063155] Key type lgssc registered [ 4487.659659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4491.311814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4491.323977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.524733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.103126] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4496.443603] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4499.695144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4499.866677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4501.642767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4505.781566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4505.927997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.557422] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4510.059527] LustreError: 313440:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.073137] LustreError: 313440:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4511.061767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4514.366606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4514.498516] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4517.100768] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4518.388769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4518.388837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4519.650258] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4522.613121] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4522.712068] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4605.411895] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4605.413838] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.421733] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4605.425218] Lustre: Skipped 1 previous similar message [ 4610.533761] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4610.537329] Lustre: Skipped 2 previous similar messages [ 4611.311176] LustreError: 314810:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4611.336300] Lustre: server umount lustre-OST0001 complete [ 4615.652249] LustreError: 312956:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.654478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.656073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4615.657680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4615.661290] LustreError: 312956:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4615.689097] Lustre: Skipped 1 previous similar message [ 4619.567198] LustreError: 315011:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4619.570968] LustreError: 315011:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4619.667150] Lustre: server umount lustre-OST0000 complete [ 4622.816426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4622.820720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.828516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4622.832128] Lustre: Skipped 1 previous similar message [ 4625.888882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.614926] LustreError: 315213:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4627.619748] LustreError: 315213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4627.679733] Lustre: server umount lustre-MDT0000 complete [ 4629.565232] LustreError: 314707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641496 with bad export cookie 3143869331759799808 [ 4629.571791] LustreError: 314707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4629.573619] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4629.669858] LustreError: 315415:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4629.672252] LustreError: 315415:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4629.794591] Lustre: server umount lustre-MDT0001 complete [ 4634.801822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4636.441232] Key type lgssc unregistered [ 4636.622717] LNet: 315943:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4636.627222] LNetError: 315943:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4636.636625] LNet: Removed LNI 192.168.203.155@tcp [ 4637.097544] Key type .llcrypt unregistered [ 4637.099557] Key type ._llcrypt unregistered [ 4659.550018] Key type ._llcrypt registered [ 4659.551947] Key type .llcrypt registered [ 4659.606054] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4659.930919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4659.962860] alg: No test for adler32 (adler32-zlib) [ 4660.837524] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4660.951389] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4662.551227] Key type lgssc registered [ 4663.083290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4667.361900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4667.374039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.581488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.274767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4674.416254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4674.610187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4676.241684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4677.638492] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4680.849280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4681.030278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.781318] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4685.165565] LustreError: 320105:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.173256] LustreError: 320105:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4688.066928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4688.219244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4690.971398] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4693.994561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4693.995201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 4700.327724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4709.615524] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4719.587541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.594575] Lustre: Skipped 2 previous similar messages [ 4719.596416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4719.600381] Lustre: Skipped 3 previous similar messages [ 4724.706048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4725.711410] LustreError: 321569:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4725.759129] Lustre: server umount lustre-MDT0000 complete [ 4727.579392] LustreError: 319276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641594 with bad export cookie 14345928614149046630 [ 4727.584976] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4727.589773] LustreError: 319276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4727.683584] LustreError: 321771:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4727.685643] LustreError: 321771:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4727.791862] Lustre: server umount lustre-MDT0001 complete [ 4740.041214] LustreError: 321971:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4740.047703] LustreError: 321971:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4740.077563] Lustre: server umount lustre-OST0000 complete [ 4752.119867] LustreError: 322174:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4752.123534] LustreError: 322174:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4752.235775] Lustre: server umount lustre-OST0001 complete [ 4759.479360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4763.980801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.113596] LustreError: 323344:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.132919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.817317] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4769.250486] LustreError: 323344:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.623734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4771.477237] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4772.820287] Lustre: 324441:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4775.826779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4775.948347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.950179] Lustre: Skipped 1 previous similar message [ 4777.902651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4780.074813] LustreError: 324797:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.169639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4784.912415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4788.975581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 4788.975921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4793.378447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4794.458121] Lustre: 326277:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4800.480215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4800.482244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.485008] Lustre: Skipped 1 previous similar message [ 4800.486884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4800.488291] Lustre: Skipped 3 previous similar messages [ 4804.579591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.579702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4804.582403] Lustre: Skipped 1 previous similar message [ 4804.583851] Lustre: Skipped 1 previous similar message [ 4806.458595] LustreError: 326382:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4806.460091] LustreError: 326382:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4806.484083] Lustre: server umount lustre-MDT0000 complete [ 4807.583107] LustreError: 326279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641674 with bad export cookie 14345928614149048317 [ 4807.586863] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4807.707583] Lustre: server umount lustre-MDT0001 complete [ 4818.866676] LustreError: 326783:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4818.868275] LustreError: 326783:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4818.880677] Lustre: server umount lustre-OST0000 complete [ 4830.243904] Lustre: server umount lustre-OST0001 complete [ 4831.276901] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 4831.728181] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 4832.210198] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 18:14:58 (1755641698) [ 4838.607646] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4841.375673] loop0: detected capacity change from 0 to 1099511627776 [ 4841.526658] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 4841.529078] *** VALIDATE xfs *** [ 4841.535341] XFS (loop0): Mounting V5 Filesystem [ 4841.543434] XFS (loop0): Ending clean mount [ 4841.930434] loop1: detected capacity change from 0 to 18691697672192 [ 4857.120890] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4864.208526] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4867.997554] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4872.369757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.406832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4878.172238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4881.571901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4884.810116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4884.850372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4884.966247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4884.975863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4885.011425] Lustre: lustre-MDT0000: new disk, initializing [ 4885.042736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.044764] Lustre: Skipped 1 previous similar message [ 4885.050614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4886.621277] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4892.153409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4892.196171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4892.253518] Lustre: 331333:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4892.295354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4892.298784] Lustre: Skipped 1 previous similar message [ 4892.350528] Lustre: lustre-MDT0001: new disk, initializing [ 4892.414681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4892.420616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4894.068585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4898.135148] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4899.904852] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4903.108818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4903.153896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4903.298110] Lustre: lustre-OST0000: new disk, initializing [ 4903.301212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4905.159365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4905.164769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4905.207450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4905.804503] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4909.528283] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4910.768210] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4910.829019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4911.932250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4911.993908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4915.680484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.681025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.682485] LustreError: Skipped 1 previous similar message [ 4915.685379] Lustre: Skipped 1 previous similar message [ 4915.686098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.689378] Lustre: Skipped 3 previous similar messages [ 4918.637103] LustreError: 334243:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4918.639963] LustreError: 334243:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4918.703975] Lustre: server umount lustre-OST0000 complete [ 4920.800692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.806366] Lustre: Skipped 2 previous similar messages [ 4925.920072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4925.921891] Lustre: Skipped 3 previous similar messages [ 4934.111091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4934.163298] Lustre: server umount lustre-MDT0000 complete [ 4935.287071] LustreError: 333069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641801 with bad export cookie 14345928614149049864 [ 4935.290392] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.291458] LustreError: 333069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4935.428047] Lustre: server umount lustre-MDT0001 complete [ 4937.888475] XFS (loop0): Unmounting Filesystem [ 4939.333290] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 18:16:45 (1755641805) [ 4941.367149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.503869] LustreError: 335466:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.525503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.527127] Lustre: Skipped 2 previous similar messages [ 4942.620354] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4945.170511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4946.382130] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4948.008765] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4949.227036] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4951.156227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4952.864438] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4954.457559] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4962.783913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4962.787359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.793250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4962.794702] Lustre: Skipped 2 previous similar messages [ 4965.857103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4967.402989] LustreError: 337789:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4967.404786] LustreError: 337789:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4967.457604] Lustre: server umount lustre-OST0000 complete [ 4970.976902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4974.676683] Lustre: server umount lustre-MDT0000 complete [ 4975.842233] LustreError: 337992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641842 with bad export cookie 14345928614149050753 [ 4975.845488] LustreError: 337992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4975.846484] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4975.980229] Lustre: server umount lustre-MDT0001 complete [ 4979.776298] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4980.690196] Key type lgssc unregistered [ 4980.815322] LNet: 338719:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4980.817384] LNetError: 338719:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4980.827281] LNet: Removed LNI 192.168.203.155@tcp [ 4981.092089] Key type .llcrypt unregistered [ 4981.093041] Key type ._llcrypt unregistered [ 4989.778052] Key type ._llcrypt registered [ 4989.778890] Key type .llcrypt registered [ 4989.809225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4990.084772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4990.091276] alg: No test for adler32 (adler32-zlib) [ 4990.927804] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4991.000959] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4992.575102] Key type lgssc registered [ 4992.882795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4995.157581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4995.163461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.307201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4997.401857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4999.924297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5000.032693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5001.187634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5003.735408] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5005.254085] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5008.465969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5008.644479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.901830] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5012.976695] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5026.273233] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.277079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5026.278599] Lustre: Skipped 1 previous similar message [ 5028.585364] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 5031.393651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5031.395214] Lustre: Skipped 1 previous similar message [ 5031.993581] LustreError: 342547:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5032.012927] Lustre: server umount lustre-MDT0001 complete [ 5032.870426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5033.700558] LustreError: 340245:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.705108] LustreError: 340245:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5035.171617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5035.203141] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5035.204671] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5035.215864] Lustre: 340239:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5035.331212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.520426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5040.613735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5041.124348] LustreError: 343618:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5041.126263] LustreError: 343618:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5041.128294] LustreError: 340242:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5041.143247] LustreError: 343621:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -17 [ 5041.147548] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5041.147548] [ 5340.913982] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 5342.694568] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.695290] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5342.701476] Lustre: Skipped 2 previous similar messages [ 5344.971362] LustreError: 354509:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5344.975137] LustreError: 354509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5345.018806] Lustre: server umount lustre-MDT0001 complete [ 5346.021369] LustreError: 340275:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.413951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5347.810636] LustreError: 340250:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.818995] LustreError: 340250:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5350.236961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5350.295431] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5350.298922] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5350.323501] Lustre: 340240:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5350.513387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5352.339943] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5356.008445] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5358.058911] LustreError: 355580:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5358.063404] LustreError: 355580:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5358.068351] LustreError: 340242:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5358.075169] LustreError: 340242:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5358.108367] LustreError: 355585:0:(obd_config.c:685:class_setup()) Device 8 already setup (type osp) [ 5358.111798] LustreError: 355585:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -17 [ 5358.116164] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.155@tcp [ 5358.116164] [ 5656.034380] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.038678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5656.041283] Lustre: Skipped 3 previous similar messages [ 5666.274352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5666.277307] Lustre: Skipped 5 previous similar messages [ 5670.367095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5670.397530] LustreError: 366569:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5670.399247] LustreError: 366569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5670.425215] Lustre: server umount lustre-MDT0001 complete [ 5671.359879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5671.397416] LustreError: 340245:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5671.401039] LustreError: 340245:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5673.729394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5673.758319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5673.759888] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5673.773220] Lustre: 340240:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5673.895247] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5675.105774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5679.084554] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5681.638515] LustreError: 367740:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5681.640710] LustreError: 367740:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5681.644647] LustreError: 340242:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5681.646301] LustreError: 340242:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5681.663473] LustreError: 367743:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 5681.668291] LustreError: 367743:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -114 [ 5681.672826] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5681.672826] [ 5991.396308] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.397827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.398751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5991.398759] Lustre: Skipped 1 previous similar message [ 5991.404862] Lustre: Skipped 2 previous similar messages [ 5995.018455] LustreError: 378697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5995.022040] LustreError: 378697:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5995.056454] Lustre: server umount lustre-MDT0000 complete [ 5996.485701] LustreError: 340231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642863 with bad export cookie 14473201453106141593 [ 5996.492265] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.494108] LustreError: 340231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5996.511983] LustreError: 345530:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.519798] LustreError: 345530:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5996.523424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.528603] Lustre: Skipped 1 previous similar message [ 6002.631971] LustreError: 378898:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6002.635320] LustreError: 378898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6002.735266] Lustre: server umount lustre-MDT0001 complete [ 6004.280427] Lustre: server umount lustre-OST0000 complete [ 6007.825652] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6012.362285] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6016.914718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.228994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.359742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6026.614113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.802523] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 18:34:56 (1755642896) [ 6032.734042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.754204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.839052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6032.853529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6032.905076] Lustre: lustre-MDT0000: new disk, initializing [ 6032.941137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.951140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6034.614245] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6040.105511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6040.136907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6040.183065] Lustre: 381620:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6040.223565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6040.226985] Lustre: Skipped 1 previous similar message [ 6040.279068] Lustre: lustre-MDT0001: new disk, initializing [ 6040.332068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6040.353531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6040.362340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6042.001107] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6046.109793] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6047.985078] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6051.373134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6051.412525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6051.567698] Lustre: lustre-OST0000: new disk, initializing [ 6051.573362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6051.580025] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6051.583324] Lustre: Skipped 4 previous similar messages [ 6051.635581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.155887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6056.945069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6056.950771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6056.975198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6058.256213] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6061.023692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6061.025666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.001754] LustreError: 384048:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 6067.003497] LustreError: 384048:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6067.032765] Lustre: server umount lustre-MDT0000 complete [ 6067.169575] LustreError: 381626:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.173387] LustreError: 381626:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6068.491828] LustreError: 381612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642935 with bad export cookie 14473201453106143546 [ 6068.497186] LustreError: 382407:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 6068.498828] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.499719] LustreError: 381612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6068.691522] Lustre: server umount lustre-MDT0001 complete [ 6080.533825] Lustre: server umount lustre-OST0000 complete [ 6093.700346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6098.412441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6103.286285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6106.880177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6109.809446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6112.526806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6113.225601] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5981 sec ========== 18:36:19 (1755642979) [ 6113.928339] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:36:20 (1755642980) === [ 6115.094265] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:36:21 (1755642981) === [ 6132.756801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6134.296463] Key type lgssc unregistered [ 6134.433614] LNet: 389536:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6134.438091] LNetError: 389536:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6134.448600] LNet: Removed LNI 192.168.203.155@tcp [ 6134.928859] Key type .llcrypt unregistered [ 6134.930852] Key type ._llcrypt unregistered