[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514195156 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002261] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007450] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009007] pid_max: default: 32768 minimum: 301 [ 0.010114] LSM: Security Framework initializing [ 0.011035] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.020751] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025062] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027068] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] *** VALIDATE tmpfs *** [ 0.030396] *** VALIDATE proc *** [ 0.031149] *** VALIDATE cgroup *** [ 0.032004] *** VALIDATE cgroup2 *** [ 0.033167] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034128] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035024] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.038577] debug: unmapping init [mem 0xffffffffb2c59000-0xffffffffb2c60fff] [ 0.041171] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042506] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043019] ... version: 2 [ 0.044008] ... bit width: 48 [ 0.044739] ... generic registers: 4 [ 0.045007] ... value mask: 0000ffffffffffff [ 0.046011] ... max period: 00007fffffffffff [ 0.047013] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049232] rcu: Hierarchical SRCU implementation. [ 0.051093] smp: Bringing up secondary CPUs ... [ 0.052442] x86: Booting SMP configuration: [ 0.053025] .... node #0, CPUs: #1 #2 #3 [ 0.056517] smp: Brought up 1 node, 4 CPUs [ 0.057810] smpboot: Max logical packages: 1 [ 0.058009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125022] node 0 deferred pages initialised in 65ms [ 0.128151] devtmpfs: initialized [ 0.129293] x86/mm: Memory block size: 128MB [ 0.131906] gcov: version magic: 0x41383552 [ 0.133217] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135324] pinctrl core: initialized pinctrl subsystem [ 0.136240] [ 0.136784] ************************************************************* [ 0.137012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138009] ** ** [ 0.139011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140012] ** ** [ 0.141025] ** This means that this kernel is built to expose internal ** [ 0.142012] ** IOMMU data structures, which may compromise security on ** [ 0.143011] ** your system. ** [ 0.144011] ** ** [ 0.145013] ** If you see this message and you are not debugging the ** [ 0.146011] ** kernel, report this immediately to your vendor! ** [ 0.147011] ** ** [ 0.148012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149013] ************************************************************* [ 0.150739] NET: Registered protocol family 16 [ 0.151423] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.152055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.154470] cpuidle: using governor menu [ 0.160226] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165634] PCI: Using configuration type 1 for base access [ 0.170338] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190394] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192320] cryptd: max_cpu_qlen set to 1000 [ 0.194187] ACPI: Added _OSI(Module Device) [ 0.195012] ACPI: Added _OSI(Processor Device) [ 0.196011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204051] ACPI: Interpreter enabled [ 0.205057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206012] ACPI: Using IOAPIC for interrupt routing [ 0.207109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217850] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222325] acpiphp: Slot [2] registered [ 0.223117] acpiphp: Slot [5] registered [ 0.224099] acpiphp: Slot [6] registered [ 0.225081] acpiphp: Slot [7] registered [ 0.226108] acpiphp: Slot [8] registered [ 0.227087] acpiphp: Slot [9] registered [ 0.228146] acpiphp: Slot [10] registered [ 0.229111] acpiphp: Slot [3] registered [ 0.230082] acpiphp: Slot [4] registered [ 0.231107] acpiphp: Slot [11] registered [ 0.232000] acpiphp: Slot [12] registered [ 0.233077] acpiphp: Slot [13] registered [ 0.234056] acpiphp: Slot [14] registered [ 0.235083] acpiphp: Slot [15] registered [ 0.237064] acpiphp: Slot [16] registered [ 0.238089] acpiphp: Slot [17] registered [ 0.240170] acpiphp: Slot [18] registered [ 0.241057] acpiphp: Slot [19] registered [ 0.242064] acpiphp: Slot [20] registered [ 0.244221] acpiphp: Slot [21] registered [ 0.245113] acpiphp: Slot [22] registered [ 0.246084] acpiphp: Slot [23] registered [ 0.247050] acpiphp: Slot [24] registered [ 0.249067] acpiphp: Slot [25] registered [ 0.250061] acpiphp: Slot [26] registered [ 0.251123] acpiphp: Slot [27] registered [ 0.253203] acpiphp: Slot [28] registered [ 0.255127] acpiphp: Slot [29] registered [ 0.257149] acpiphp: Slot [30] registered [ 0.259288] acpiphp: Slot [31] registered [ 0.262152] PCI host bridge to bus 0000:00 [ 0.264024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.267035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273097] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.276027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.280015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.285779] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.289148] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.303012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.305013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.308011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.309490] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.311577] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.315657] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.318852] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.330016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.333016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.339750] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.349016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.355012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.372014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.385548] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.394017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.401016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.416014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427381] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.434016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.448039] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.476022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.496120] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.509018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.559015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.571538] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.618019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.634788] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.667017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.707018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.727270] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.728377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.731332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.734428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.736156] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.741056] iommu: Default domain type: Passthrough [ 0.742477] SCSI subsystem initialized [ 0.743185] ACPI: bus type USB registered [ 0.745056] usbcore: registered new interface driver usbfs [ 0.746063] usbcore: registered new interface driver hub [ 0.747058] usbcore: registered new device driver usb [ 0.749140] pps_core: LinuxPPS API ver. 1 registered [ 0.751005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752070] PTP clock support registered [ 0.754146] EDAC MC: Ver: 3.0.0 [ 0.755509] PCI: Using ACPI for IRQ routing [ 0.757126] NetLabel: Initializing [ 0.759010] NetLabel: domain hash size = 128 [ 0.761011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.764086] NetLabel: unlabeled traffic allowed by default [ 0.766132] vgaarb: loaded [ 0.767448] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.777189] clocksource: Switched to clocksource kvm-clock [ 0.880932] VFS: Disk quotas dquot_6.6.0 [ 0.882671] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.885393] *** VALIDATE ramfs *** [ 0.886641] *** VALIDATE hugetlbfs *** [ 0.888223] pnp: PnP ACPI init [ 0.891027] pnp: PnP ACPI: found 6 devices [ 0.911385] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.914758] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.917079] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.918842] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.921042] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.923620] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.926531] NET: Registered protocol family 2 [ 0.929181] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.934140] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.937831] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.943249] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.946879] TCP: Hash tables configured (established 65536 bind 65536) [ 0.949670] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.951757] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955650] NET: Registered protocol family 1 [ 0.957364] RPC: Registered named UNIX socket transport module. [ 0.958775] RPC: Registered udp transport module. [ 0.959839] RPC: Registered tcp transport module. [ 0.960929] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.962367] NET: Registered protocol family 44 [ 0.963456] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964972] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966259] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.967750] PCI: CLS 0 bytes, default 64 [ 0.968657] Unpacking initramfs... [ 2.299490] debug: unmapping init [mem 0xffff8c907cc54000-0xffff8c907ffbffff] [ 2.303669] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.305494] software IO TLB: mapped [mem 0x00000000a1a00000-0x00000000a5a00000] (64MB) [ 2.308292] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.813138] Initialise system trusted keyrings [ 2.814171] Key type blacklist registered [ 2.815213] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.820802] zbud: loaded [ 2.822734] *** VALIDATE nfs *** [ 2.823623] *** VALIDATE nfs4 *** [ 2.824507] pstore: using deflate compression [ 2.826594] Platform Keyring initialized [ 2.936047] NET: Registered protocol family 38 [ 2.937960] Key type asymmetric registered [ 2.940341] Asymmetric key parser 'x509' registered [ 2.943129] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.947825] io scheduler mq-deadline registered [ 2.950078] io scheduler kyber registered [ 2.951812] io scheduler bfq registered [ 2.954778] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.958521] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.961519] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.965123] ACPI: Power Button [PWRF] [ 2.971990] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.978741] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.993172] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.999404] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.012887] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.042867] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.073551] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.078347] Non-volatile memory driver v1.3 [ 3.079883] Linux agpgart interface v0.103 [ 3.111525] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.114734] vda: detected capacity change from 0 to 69627904 [ 3.128503] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.131570] vdb: detected capacity change from 0 to 1073741824 [ 3.157658] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.162970] vdc: detected capacity change from 0 to 2621440000 [ 3.176967] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.180085] vdd: detected capacity change from 0 to 2621440000 [ 3.197140] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.200412] vde: detected capacity change from 0 to 4294967296 [ 3.222894] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.226665] vdf: detected capacity change from 0 to 4294967296 [ 3.236622] libphy: Fixed MDIO Bus: probed [ 3.243385] usbcore: registered new interface driver usbserial_generic [ 3.246233] usbserial: USB Serial support registered for generic [ 3.248538] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.253549] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.255449] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.257613] mousedev: PS/2 mouse device common for all mice [ 3.261967] rtc_cmos 00:05: RTC can wake from S4 [ 3.264694] rtc_cmos 00:05: registered as rtc0 [ 3.267583] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.268551] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.269616] intel_pstate: CPU model not supported [ 3.274963] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.278165] hid: raw HID events driver (C) Jiri Kosina [ 3.279945] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.281195] usbcore: registered new interface driver usbhid [ 3.285400] usbhid: USB HID core driver [ 3.286862] drop_monitor: Initializing network drop monitor service [ 3.289153] Initializing XFRM netlink socket [ 3.291046] NET: Registered protocol family 10 [ 3.295023] Segment Routing with IPv6 [ 3.296397] NET: Registered protocol family 17 [ 3.298569] mpls_gso: MPLS GSO support [ 3.304084] RAS: Correctable Errors collector initialized. [ 3.305994] AVX version of gcm_enc/dec engaged. [ 3.307144] AES CTR mode by8 optimization enabled [ 3.384539] sched_clock: Marking stable (3384465104, 0)->(4294509688, -910044584) [ 3.388664] registered taskstats version 1 [ 3.395196] Loading compiled-in X.509 certificates [ 3.397363] zswap: loaded using pool lzo/zbud [ 3.420815] Key type big_key registered [ 3.433019] Key type encrypted registered [ 3.434763] ima: No TPM chip found, activating TPM-bypass! [ 3.437059] ima: Allocated hash algorithm: sha1 [ 3.438636] ima: No architecture policies found [ 3.440271] evm: Initialising EVM extended attributes: [ 3.442108] evm: security.selinux [ 3.445091] evm: security.ima [ 3.446126] evm: security.capability [ 3.447585] evm: HMAC attrs: 0x1 [ 3.451309] rtc_cmos 00:05: setting system clock to 2026-06-02 06:08:13 UTC (1780380493) [ 3.460523] debug: unmapping init [mem 0xffffffffb3c03000-0xffffffffb3dfffff] [ 3.465949] debug: unmapping init [mem 0xffffffffb2982000-0xffffffffb2c58fff] [ 3.476219] Write protecting the kernel read-only data: 28672k [ 3.481497] debug: unmapping init [mem 0xffffffffb1003000-0xffffffffb11fffff] [ 3.484537] debug: unmapping init [mem 0xffffffffb1914000-0xffffffffb19fffff] [ 3.521333] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.530361] systemd[1]: Detected virtualization kvm. [ 3.532179] systemd[1]: Detected architecture x86-64. [ 3.534085] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.562652] systemd[1]: No hostname configured. [ 3.564733] systemd[1]: Set hostname to . [ 3.567173] random: systemd: uninitialized urandom read (16 bytes read) [ 3.572532] systemd[1]: Initializing machine ID from random generator. [ 3.743946] random: systemd: uninitialized urandom read (16 bytes read) [ 3.746781] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.751250] random: systemd: uninitialized urandom read (16 bytes read) [ 3.754401] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.758641] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.406465] device-mapper: uevent: version 1.0.3 [ 4.408204] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.190371] random: fast init done [ 5.191580] virtio_net virtio0 ens2: renamed from eth0 [ 5.496160] scsi host0: ata_piix [ 5.533652] scsi host1: ata_piix [ 5.535672] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.538721] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.973763] random: crng init done [ 9.975804] random: 7 urandom warning(s) missed due to ratelimiting [ 10.019671] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.694777] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.919713] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.204275] SELinux: Disabled at runtime. [ 12.263132] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.272849] systemd[1]: Detected virtualization kvm. [ 12.274463] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.807114] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.811248] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.820140] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.827364] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.829589] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.838421] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.847260] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.971551] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 13.109089] hrtimer: interrupt took 2797661 ns [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.373171] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.690319] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.870912] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.998866] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.036061] EDAC sbridge: Ver: 1.1.2 [ 15.995841] Key type dns_resolver registered [ 16.334184] NFS: Registering the id_resolver key type [ 16.336548] Key type id_resolver registered [ 16.338592] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg220-server login: [ 50.859753] libcfs: loading out-of-tree module taints kernel. [ 50.876417] Key type ._llcrypt registered [ 50.877337] Key type .llcrypt registered [ 50.920837] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 59.995665] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 60.717515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.727901] alg: No test for adler32 (adler32-zlib) [ 61.792588] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 62.192654] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 63.831307] Key type lgssc registered [ 64.721694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.540273] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.116482] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 105.304088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 105.320248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 106.492752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 106.516756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 106.581462] Lustre: lustre-MDT0000: new disk, initializing [ 106.647128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.674208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 108.758982] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.310400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.495134] Lustre: 6479:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 123.529716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 123.540621] Lustre: Skipped 1 previous similar message [ 123.669540] Lustre: lustre-MDT0001: new disk, initializing [ 123.833144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.880291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 123.899746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 129.888096] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.858592] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 151.588148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 151.990566] Lustre: lustre-OST0000: new disk, initializing [ 151.994795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 152.002386] Lustre: 8384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 152.101637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.758065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 157.767962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 157.857124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 160.009790] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.050849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.128635] Lustre: lustre-OST0001: new disk, initializing [ 176.130874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 176.134770] Lustre: 9441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.189844] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.222749] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.337915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.345538] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.389181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 194.795871] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 205.373987] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 211.929847] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing check_logdir /tmp/testlogs/ [ 215.987383] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing yml_node [ 219.448195] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 221.697000] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 223.919410] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 225.559386] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 2 02:11:54 EDT 2026 [ 245.196539] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 246.469963] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 261.095216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.104219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 261.110821] Lustre: Skipped 1 previous similar message [ 266.209940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 266.221837] Lustre: Skipped 6 previous similar messages [ 267.196443] Lustre: server umount lustre-MDT0000 complete [ 270.987811] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780380761 with bad export cookie 16138049406714425904 [ 270.992967] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.003984] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.328709] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.338057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 271.340438] Lustre: Skipped 3 previous similar messages [ 271.354618] Lustre: Skipped 1 previous similar message [ 275.360113] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.366347] Lustre: Skipped 1 previous similar message [ 277.477818] Lustre: server umount lustre-MDT0001 complete [ 281.075226] Lustre: server umount lustre-OST0000 complete [ 285.002854] Lustre: server umount lustre-OST0001 complete [ 291.181534] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 300.959149] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 332.305673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.616624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 332.665815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.751261] Lustre: lustre-MDT0000: new disk, initializing [ 332.891073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.948046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 337.054518] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 347.181448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.283534] Lustre: 14949:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 347.327562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 347.330326] Lustre: Skipped 1 previous similar message [ 347.385719] Lustre: lustre-MDT0001: new disk, initializing [ 347.530628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 347.560067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 347.576037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 354.567339] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 365.995473] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 371.308907] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 377.723033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 378.042884] Lustre: lustre-OST0000: new disk, initializing [ 378.046096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 378.055939] Lustre: 16699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 378.153101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.142322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 379.167687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 379.331751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 383.976435] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 392.728776] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 398.629866] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 398.842392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 404.367715] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 404.635747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 410.080488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 410.085141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.096881] LustreError: Skipped 1 previous similar message [ 410.114072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.325613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.335223] Lustre: Skipped 4 previous similar messages [ 421.343172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.832167] Lustre: server umount lustre-OST0000 complete [ 429.543895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 429.557209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.578115] Lustre: Skipped 1 previous similar message [ 432.643737] Lustre: server umount lustre-MDT0000 complete [ 435.687158] LustreError: 16292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.728281] LustreError: 16292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 436.333771] LustreError: 14941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780380926 with bad export cookie 16138049406714427444 [ 436.340355] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 436.342984] LustreError: 14941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 436.610916] Lustre: server umount lustre-MDT0001 complete [ 441.628025] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 02:15:30 (1780380930) [ 448.528394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.044414] LustreError: 18819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.086769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.873234] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 454.118529] LustreError: 18820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.232311] LustreError: 18819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.991341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.016462] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 473.956221] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 478.283182] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 485.685039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.588108] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 501.864256] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 505.450404] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 514.529318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.543659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.564509] Lustre: Skipped 1 previous similar message [ 514.572327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.580615] Lustre: Skipped 3 previous similar messages [ 518.622701] Lustre: server umount lustre-MDT0000 complete [ 520.167376] LustreError: 19789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.187180] LustreError: 19789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 522.677721] LustreError: 18801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381012 with bad export cookie 16138049406714428326 [ 522.697864] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.701915] LustreError: 18801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 523.224472] Lustre: server umount lustre-MDT0001 complete [ 538.423005] Lustre: server umount lustre-OST0000 complete [ 563.056547] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 573.402705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.789941] LustreError: 22844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.872102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.877627] Lustre: Skipped 2 previous similar messages [ 578.672937] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 587.511432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.889505] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 595.785939] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 602.757572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.978525] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 625.486357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 625.646627] Lustre: lustre-OST0001: new disk, initializing [ 625.649720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 625.655841] Lustre: 25110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 627.694036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 627.711214] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 627.824243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 633.593517] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 644.366757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 648.887929] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 663.549793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.568245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.575270] Lustre: Skipped 4 previous similar messages [ 663.592069] Lustre: Skipped 6 previous similar messages [ 668.095650] Lustre: server umount lustre-MDT0000 complete [ 668.648346] LustreError: 24545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.680102] LustreError: 24545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 672.090779] LustreError: 25108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381162 with bad export cookie 16138049406714429173 [ 672.099590] LustreError: 25108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 672.100140] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.471581] Lustre: server umount lustre-MDT0001 complete [ 687.372365] Lustre: server umount lustre-OST0000 complete [ 689.632575] Lustre: 3637:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780381163/real 1780381163] req@ffff8c90fbae0a80 x1866864317942016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780381179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 689.690303] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.726216] Lustre: Skipped 1 previous similar message [ 691.001434] Lustre: server umount lustre-OST0001 complete [ 709.277504] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 723.110238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.778349] LustreError: 28100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.927400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.937248] Lustre: Skipped 3 previous similar messages [ 730.021456] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 740.390947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.808334] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 747.992430] Lustre: 29213:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 754.570134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 761.500974] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 770.542784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 776.847992] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 784.729129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 788.006460] Lustre: 31058:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 796.653140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.659774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.673930] Lustre: Skipped 2 previous similar messages [ 796.677558] Lustre: Skipped 1 previous similar message [ 799.558309] Lustre: server umount lustre-MDT0000 complete [ 801.785710] LustreError: 28094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.829305] LustreError: 28094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 803.440791] LustreError: 28081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381293 with bad export cookie 16138049406714430468 [ 803.446604] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 803.454424] LustreError: 28081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 804.017618] Lustre: server umount lustre-MDT0001 complete [ 818.278052] Lustre: server umount lustre-OST0000 complete [ 822.254861] Lustre: 3636:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780381296/real 1780381296] req@ffff8c90c5b9f800 x1866864317990784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780381312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 822.801737] Lustre: server umount lustre-OST0001 complete [ 833.694219] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 02:22:02 (1780381322) [ 863.676829] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 873.389833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.701547] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 885.291131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.067661] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 896.257850] Lustre: 34955:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 903.104852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 908.660394] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 917.217272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.541032] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 931.191671] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 934.729412] Lustre: 36801:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 942.049179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.056579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.082729] Lustre: Skipped 2 previous similar messages [ 942.086465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 945.591960] Lustre: server umount lustre-MDT0000 complete [ 949.224779] LustreError: 34765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.250732] LustreError: 34765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 949.374742] LustreError: 33822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381439 with bad export cookie 16138049406714431574 [ 949.383832] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.396167] LustreError: 33822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 949.707665] Lustre: server umount lustre-MDT0001 complete [ 963.859940] Lustre: server umount lustre-OST0000 complete [ 978.816795] Lustre: server umount lustre-OST0001 complete [ 985.645348] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 02:24:35 (1780381475) [ 997.861959] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1006.074907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.527506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.532799] Lustre: Skipped 7 previous similar messages [ 1010.650532] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1018.074778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.690742] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1025.632684] Lustre: 40042:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1034.365450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.909706] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1052.375712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.929642] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1066.186432] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1069.704578] Lustre: 41886:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1070.510957] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1070.515730] Lustre: Skipped 1 previous similar message [ 1074.659751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1074.666750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.674050] Lustre: Skipped 3 previous similar messages [ 1080.578681] Lustre: server umount lustre-MDT0000 complete [ 1085.378994] LustreError: 38909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381575 with bad export cookie 16138049406714432680 [ 1085.389027] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1085.402228] LustreError: 38909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1091.983474] Lustre: server umount lustre-MDT0001 complete [ 1100.984082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.502037] LustreError: 3633:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c90f2d98380 x1866864318094848/t0(0) o250->MGC192.168.202.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1115.677952] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1121.270034] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1123.730193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.819330] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1129.453094] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1129.462931] Lustre: Skipped 1 previous similar message [ 1136.079690] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1138.155692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1142.421687] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1144.612956] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1149.929587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.181507] Lustre: server umount lustre-MDT0000 complete [ 1160.602073] LustreError: 38910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381650 with bad export cookie 16138049406714433807 [ 1160.613568] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.045539] Lustre: server umount lustre-MDT0001 complete [ 1175.804709] Lustre: server umount lustre-OST0000 complete [ 1191.628294] Lustre: server umount lustre-OST0001 complete [ 1203.345479] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 02:28:12 (1780381692) [ 1222.266478] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1231.942377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.301154] LustreError: 46253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.315874] LustreError: 46253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 32 previous similar messages [ 1236.371245] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1246.092867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.854425] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1258.027509] Lustre: 47373:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1266.454935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1275.143694] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1285.710306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1293.153765] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1300.992740] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1305.641965] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1318.884956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.892978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.899320] Lustre: Skipped 23 previous similar messages [ 1319.860411] Lustre: server umount lustre-MDT0000 complete [ 1324.020425] LustreError: 46234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381814 with bad export cookie 16138049406714434598 [ 1324.028310] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.035112] LustreError: 46234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1324.439357] Lustre: server umount lustre-MDT0001 complete [ 1339.126692] Lustre: server umount lustre-OST0000 complete [ 1353.678370] Lustre: server umount lustre-OST0001 complete [ 1362.814869] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 02:30:52 (1780381852) [ 1365.379534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1367.368442] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 02:30:56 (1780381856) [ 1380.621696] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 1391.866755] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1414.363034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.791489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1414.883440] Lustre: lustre-MDT0000: new disk, initializing [ 1415.018560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1419.477401] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1427.594732] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1434.485917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1435.090879] Lustre: lustre-OST0000: new disk, initializing [ 1435.100552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1435.111930] Lustre: Skipped 1 previous similar message [ 1435.122198] Lustre: 53270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1436.662965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1436.683494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1436.728241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1442.073371] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1449.852164] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1452.634809] Lustre: DEBUG MARKER: create test files [ 1482.725209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1482.741354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.761460] Lustre: Skipped 13 previous similar messages [ 1488.732478] Lustre: server umount lustre-OST0000 complete [ 1492.606462] Lustre: server umount lustre-MDT0000 complete [ 1504.381750] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1508.075164] Key type lgssc unregistered [ 1508.445130] LNet: 54771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.459626] LNetError: 54771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.483988] LNet: Removed LNI 192.168.202.120@tcp [ 1509.761167] Key type .llcrypt unregistered [ 1509.763480] Key type ._llcrypt unregistered [ 1542.669931] Key type ._llcrypt registered [ 1542.672979] Key type .llcrypt registered [ 1542.790642] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1543.990780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.510554] alg: No test for adler32 (adler32-zlib) [ 1545.808650] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1546.188207] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1547.847235] Key type lgssc registered [ 1549.037421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.974281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1559.024056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.755633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.825329] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1572.769119] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1581.037588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.320194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.490713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:97 to 0x240000400:129) [ 1585.628888] Lustre: 57377:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1585.632161] Lustre: 57377:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1585.639251] Lustre: 57377:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1585.646662] Lustre: 57377:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1585.654549] Lustre: 57377:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1585.662718] Lustre: 57377:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1587.108037] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1594.016389] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1599.381408] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1604.620957] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1611.239561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1611.241902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.271678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.379517] Lustre: server umount lustre-OST0000 complete [ 1617.521748] Lustre: server umount lustre-MDT0000 complete [ 1634.737719] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1638.320872] Key type lgssc unregistered [ 1638.637049] LNet: 58856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.652058] LNetError: 58856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.672513] LNet: Removed LNI 192.168.202.120@tcp [ 1639.469673] Key type .llcrypt unregistered [ 1639.472302] Key type ._llcrypt unregistered [ 1676.017668] Key type ._llcrypt registered [ 1676.019427] Key type .llcrypt registered [ 1676.156847] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1677.422875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.481134] alg: No test for adler32 (adler32-zlib) [ 1678.788078] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1678.994543] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1680.712473] Key type lgssc registered [ 1681.770897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1689.622569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1689.651378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.253833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.311937] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1702.120090] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1709.552639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.868822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.052754] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1716.080878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1716.278886] Lustre: 61562:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1716.299369] Lustre: 61562:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1716.311974] Lustre: 61562:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1716.322704] Lustre: 61562:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1716.332179] Lustre: 61562:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1716.340582] Lustre: 61562:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1721.837958] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1728.260068] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1745.658819] Lustre: server umount lustre-OST0000 complete [ 1761.568763] Lustre: server umount lustre-MDT0000 complete [ 1772.718866] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 1776.294499] Key type lgssc unregistered [ 1776.699912] LNet: 63011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.712251] LNetError: 63011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.775861] LNet: Removed LNI 192.168.202.120@tcp [ 1778.708582] Key type .llcrypt unregistered [ 1778.710563] Key type ._llcrypt unregistered [ 1799.725125] Key type ._llcrypt registered [ 1799.729270] Key type .llcrypt registered [ 1799.861555] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 1816.615208] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 1818.030075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.053728] alg: No test for adler32 (adler32-zlib) [ 1819.134629] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1819.299780] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 1820.968511] Key type lgssc registered [ 1822.255468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.822930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.864989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.224930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1861.285959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1861.422225] Lustre: lustre-MDT0000: new disk, initializing [ 1861.543696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.586450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1867.857956] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1880.657863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.891888] Lustre: 66133:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1881.044351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1881.057701] Lustre: Skipped 1 previous similar message [ 1881.230617] Lustre: lustre-MDT0001: new disk, initializing [ 1881.439816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1881.482166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1881.501298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1884.997300] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1893.309984] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1898.148479] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1905.038211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1905.415392] Lustre: lustre-OST0000: new disk, initializing [ 1905.425154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1905.433520] Lustre: 67875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1905.610585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.924167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1911.930718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1912.100494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1914.427962] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 1922.601254] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1927.719673] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1928.063751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1931.916529] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1932.178453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1937.892776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1937.902377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.910072] LustreError: Skipped 1 previous similar message [ 1937.911246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1937.938227] Lustre: Skipped 1 previous similar message [ 1940.526156] Lustre: server umount lustre-OST0000 complete [ 1948.129078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.131679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.146651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.155536] Lustre: Skipped 2 previous similar messages [ 1950.877083] Lustre: server umount lustre-MDT0000 complete [ 1954.849785] LustreError: 66125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382444 with bad export cookie 7726810763858285682 [ 1954.859291] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.875407] LustreError: 66125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.268745] Lustre: server umount lustre-MDT0001 complete [ 1965.099501] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 02:40:54 (1780382454) [ 1977.540205] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2021.432189] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2039.994121] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2086.370099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.654279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2086.673597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2086.718800] Lustre: lustre-MDT0000: new disk, initializing [ 2086.774461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.795373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2091.219426] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2104.390164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.580788] Lustre: 73722:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2104.679918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2104.695412] Lustre: Skipped 1 previous similar message [ 2104.874576] Lustre: lustre-MDT0001: new disk, initializing [ 2105.177025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.251960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2105.276347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2112.635995] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2124.529884] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2128.485360] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2135.684480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.214201] Lustre: lustre-OST0000: new disk, initializing [ 2136.221427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2136.233360] Lustre: 75466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2136.398513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.298235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2143.316781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2143.449948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2143.665516] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2152.358182] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2157.057516] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2157.414242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2162.873317] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2163.114194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2169.313837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.320872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2169.321236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.353570] Lustre: Skipped 2 previous similar messages [ 2170.948681] Lustre: server umount lustre-OST0000 complete [ 2176.995089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.005159] LustreError: Skipped 1 previous similar message [ 2177.008586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.023870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.026644] Lustre: Skipped 1 previous similar message [ 2180.868646] Lustre: server umount lustre-MDT0000 complete [ 2184.673265] LustreError: 73734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.702082] LustreError: 73734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2186.613771] LustreError: 73715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382676 with bad export cookie 7726810763858286564 [ 2186.617298] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.625671] LustreError: 73715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2187.304736] Lustre: server umount lustre-MDT0001 complete [ 2197.708559] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 02:44:46 (1780382686) [ 2205.556680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.178562] LustreError: 77730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.311893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.945637] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2211.305563] LustreError: 77731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.420287] LustreError: 77730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.815365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.955482] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2230.732266] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2234.938743] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2242.049263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.481338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.490358] Lustre: Skipped 1 previous similar message [ 2248.801564] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2254.397492] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2257.861191] Lustre: 80059:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 1786eaad-38a6-4b49-9718-629eac58af5a at adminstrative request [ 2262.008712] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.026984] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2272.237033] Lustre: MGS: Client 1786eaad-38a6-4b49-9718-629eac58af5a (at 0@lo) reconnecting [ 2272.250078] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x6b3b25738a701b56 to 0x6b3b25738a701df6 [ 2272.287070] Lustre: MGC192.168.202.120@tcp: Connection restored to 0@lo (at 0@lo) [ 2297.640070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.919701] Lustre: lustre-OST0001: new disk, initializing [ 2297.930430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2297.946265] Lustre: 80453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2303.679964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2303.699217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2303.894163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2306.344442] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2316.070111] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2324.448885] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.454161] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.459358] LustreError: Skipped 1 previous similar message [ 2324.466716] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2324.469539] Lustre: Skipped 2 previous similar messages [ 2324.471901] Lustre: Skipped 2 previous similar messages [ 2325.841863] Lustre: server umount lustre-OST0001 complete [ 2329.574345] LustreError: 81012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.593988] LustreError: 81012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2332.640963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2332.647137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.667260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2332.678524] Lustre: Skipped 1 previous similar message [ 2336.535892] Lustre: server umount lustre-OST0000 complete [ 2347.550111] Lustre: server umount lustre-MDT0000 complete [ 2350.050451] LustreError: 77731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.059887] LustreError: 77731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2351.687311] LustreError: 80463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382841 with bad export cookie 7726810763858288118 [ 2351.691223] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2351.706226] LustreError: 80463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2352.357941] Lustre: server umount lustre-MDT0001 complete [ 2363.730852] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 02:47:32 (1780382852) [ 2366.004923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2368.162764] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 02:47:36 (1780382856) [ 2370.110567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2372.013812] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 02:47:41 (1780382861) [ 2374.319170] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2376.025845] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 02:47:45 (1780382865) [ 2378.236389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2380.127731] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 02:47:49 (1780382869) [ 2381.922639] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2388.751750] loop: module loaded [ 2388.770984] loop0: detected capacity change from 0 to 4294967296 [ 2388.839969] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2390.829727] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2395.012985] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2399.550407] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2428.557556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.607359] LustreError: 84449:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2428.611844] LustreError: 84449:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2428.617358] LustreError: 84449:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2428.624241] LustreError: 84449:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2447.597637] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 2458.788792] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 2499.715303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.091626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2500.123750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2500.194107] Lustre: lustre-MDT0000: new disk, initializing [ 2500.287198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.296453] Lustre: Skipped 1 previous similar message [ 2500.308810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2505.972662] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2517.580945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.754449] Lustre: 87283:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2517.973947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2523.661181] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2533.455811] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.717257] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2543.742047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2544.191715] Lustre: 89026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2549.863188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2550.040593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2550.235434] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2557.979047] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2563.905415] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2564.206176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2568.929800] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2569.180916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2575.850595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.858154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.878260] Lustre: Skipped 3 previous similar messages [ 2575.886057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.886067] Lustre: Skipped 6 previous similar messages [ 2575.926373] LustreError: Skipped 3 previous similar messages [ 2577.288920] Lustre: server umount lustre-OST0000 complete [ 2587.783545] Lustre: server umount lustre-MDT0000 complete [ 2591.207861] LustreError: 87289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.240336] LustreError: 87289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2591.640344] LustreError: 87276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383081 with bad export cookie 7726810763858288734 [ 2591.644960] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2591.653759] LustreError: 87276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2592.057618] Lustre: server umount lustre-MDT0001 complete [ 2602.285077] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 02:51:30 (1780383090) [ 2610.877544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.732650] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2626.532344] LustreError: 91289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.552063] LustreError: 91289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2628.572980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.966762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2628.997050] Lustre: Skipped 3 previous similar messages [ 2633.246483] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2639.736450] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2644.492630] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2652.531620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.914036] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2666.615788] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.040837] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2671.217217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2674.875463] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2675.059419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2681.980519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2682.168560] Lustre: lustre-OST0001: new disk, initializing [ 2682.182016] Lustre: Skipped 2 previous similar messages [ 2682.188855] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2682.196559] Lustre: Skipped 3 previous similar messages [ 2682.202728] Lustre: 94283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2688.289419] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2689.125686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2689.135941] Lustre: Skipped 2 previous similar messages [ 2689.150827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2689.319843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2695.710159] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2701.380846] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2701.772721] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2708.537453] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2709.025790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2731.208691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2757.841746] Lustre: Failing over lustre-MDT0000 [ 2757.953648] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 2757.968204] Lustre: Skipped 5 previous similar messages [ 2758.554199] Lustre: server umount lustre-MDT0000 complete [ 2759.137360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.162372] LustreError: 91283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.167101] Lustre: Skipped 5 previous similar messages [ 2759.218409] LustreError: 91283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2775.455221] Lustre: 64357:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780383249/real 1780383249] req@000000005de1ca29 x1866866160319616/t0(0) o400->MGC192.168.202.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780383265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2775.491783] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2785.737796] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2785.739988] LDISKFS-fs (dm-0): recovery complete [ 2785.746741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.649699] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2805.025315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2805.025343] Lustre: 95391:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2805.436526] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 2805.733913] Lustre: 91288:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2805.743126] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2805.750126] Lustre: 91288:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 1 previous similar message [ 2805.914705] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2806.239592] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2806.569593] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2806.586897] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 1 previous similar message [ 2806.917442] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2806.928910] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 1 previous similar message [ 2806.958210] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2806.975615] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 7 previous similar messages [ 2807.632266] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2807.642504] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 2 previous similar messages [ 2807.985147] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2807.996318] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 2 previous similar messages [ 2808.968102] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2808.985559] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 5 previous similar messages [ 2809.947466] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2809.964090] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 6 previous similar messages [ 2810.289131] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2810.297255] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 6 previous similar messages [ 2813.163200] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2813.175086] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 12 previous similar messages [ 2814.136116] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2814.150337] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 12 previous similar messages [ 2814.471173] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2814.474809] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 12 previous similar messages [ 2821.168585] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2821.196094] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 24 previous similar messages [ 2822.155479] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2822.168356] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 24 previous similar messages [ 2822.495155] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2822.512226] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 24 previous similar messages [ 2837.208159] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2837.229965] Lustre: 96988:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 49 previous similar messages [ 2838.193896] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2838.213932] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 49 previous similar messages [ 2838.540248] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2838.550736] LustreError: 96988:0:(ldlm_lib.c:2682:replay_request_or_update()) Skipped 49 previous similar messages [ 2865.207161] Lustre: 96988:0:(ldlm_lib.c:2378:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2865.218448] Lustre: 96988:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2865.239662] Lustre: 96988:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a2f26904-3bc9-455b-a57c-91733d55b85d@192.168.202.20@tcp [ 2865.257707] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2865.271050] Lustre: 96988:0:(ldlm_lib.c:1898:abort_req_replay_queue()) @@@ aborted: req@000000007715cddd x1866866143337344/t0(8589934782) o35->a2f26904-3bc9-455b-a57c-91733d55b85d@192.168.202.20@tcp:500/0 lens 392/0 e 0 to 0 dl 1780383365 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2865.297532] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2865.316810] Lustre: lustre-MDT0000: Denying connection for new client a2f26904-3bc9-455b-a57c-91733d55b85d (at 192.168.202.20@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2865.321886] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2865.389102] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2865.405040] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2865.406226] Lustre: Skipped 2 previous similar messages [ 2865.489528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:81 to 0x280000401:577) [ 2865.489980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2884.577704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2884.598900] LustreError: Skipped 1 previous similar message [ 2884.611160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.643618] Lustre: Skipped 1 previous similar message [ 2884.647808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.375237] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2897.460890] Lustre: server umount lustre-OST0000 complete [ 2897.895889] LustreError: 92883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.928059] LustreError: 92883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 59 previous similar messages [ 2908.606584] Lustre: server umount lustre-OST0001 complete [ 2919.340494] Lustre: server umount lustre-MDT0000 complete [ 2923.379700] LustreError: 92006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383413 with bad export cookie 7726810763858340002 [ 2923.381136] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.397945] LustreError: 92006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2923.804740] Lustre: server umount lustre-MDT0001 complete [ 2932.731299] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 02:57:01 (1780383421) [ 2940.661514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.674191] Lustre: *** cfs_fail_loc=197, val=0*** [ 2956.384336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2961.503426] LustreError: 98949:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 2966.648792] Lustre: *** cfs_fail_loc=197, val=0*** [ 2966.652104] Lustre: Skipped 65 previous similar messages [ 2987.231353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.242478] Lustre: Skipped 3 previous similar messages [ 2994.755259] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3001.062177] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3003.513976] Lustre: server umount lustre-OST0000 complete [ 3013.539662] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 02:58:22 (1780383502) [ 3035.669989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.067731] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3048.605151] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3052.183065] LustreError: 100488:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3052.191858] LustreError: 100488:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3052.197598] LustreError: 100488:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 3052.452087] Lustre: server umount lustre-MDT0000 complete [ 3097.957950] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3111.066946] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3150.997805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.344187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3151.406189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3151.547322] Lustre: lustre-MDT0000: new disk, initializing [ 3151.689815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3156.897879] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3168.224512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.349286] Lustre: 104517:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3168.620481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3174.133176] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3182.233225] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3186.061603] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3193.515741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.927109] Lustre: 106272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3200.189776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3200.509991] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3208.983422] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3213.999572] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3214.216821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.444678] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3219.788316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3226.080417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3226.095180] LustreError: Skipped 1 previous similar message [ 3226.098975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.128839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3226.140358] Lustre: Skipped 6 previous similar messages [ 3226.159069] Lustre: Skipped 14 previous similar messages [ 3228.946414] Lustre: server umount lustre-OST0000 complete [ 3239.254377] Lustre: server umount lustre-MDT0000 complete [ 3241.461299] LustreError: 106569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.493216] LustreError: 106569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3243.226918] LustreError: 104510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383733 with bad export cookie 7726810763858350677 [ 3243.232782] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.235595] LustreError: 104510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3243.659513] Lustre: server umount lustre-MDT0001 complete [ 3253.161709] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 03:02:22 (1780383742) [ 3262.877356] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3265.706779] Key type lgssc unregistered [ 3266.176599] LNet: 108594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3266.182530] LNetError: 108594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3266.199268] LNet: Removed LNI 192.168.202.120@tcp [ 3267.173929] Key type .llcrypt unregistered [ 3267.176102] Key type ._llcrypt unregistered [ 3284.598388] Key type ._llcrypt registered [ 3284.602275] Key type .llcrypt registered [ 3284.783820] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3298.108909] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3299.265193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3299.295140] alg: No test for adler32 (adler32-zlib) [ 3300.617244] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3300.949226] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3302.767184] Key type lgssc registered [ 3304.081771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3347.323500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3347.367304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.695104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3348.735739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3348.842264] Lustre: lustre-MDT0000: new disk, initializing [ 3348.964486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.005634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3353.760655] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3365.658616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.123414] Lustre: lustre-OST0000: new disk, initializing [ 3366.129291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3366.146335] Lustre: Skipped 1 previous similar message [ 3366.157316] Lustre: 112719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3366.263690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.247783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3372.255334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3372.307563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3373.332093] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3384.472985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.737315] Lustre: lustre-OST0001: new disk, initializing [ 3384.740592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3384.746163] Lustre: 113564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3384.820058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.261561] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3394.721297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3394.730830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3394.753465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3400.162144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.196246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.207518] Lustre: Skipped 1 previous similar message [ 3405.281323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.292326] Lustre: Skipped 1 previous similar message [ 3405.928188] Lustre: server umount lustre-MDT0000 complete [ 3413.124542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3418.282177] LustreError: 111922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383908 with bad export cookie 436244792367190094 [ 3418.299610] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3418.370658] Lustre: server umount lustre-OST0000 complete [ 3422.368140] Lustre: server umount lustre-OST0001 complete [ 3432.324305] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 03:05:21 (1780383921) [ 3443.300947] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3466.666715] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 03:05:55 (1780383955) [ 3510.719224] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 3522.420500] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3559.640847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3560.105640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3560.150291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3560.235918] Lustre: lustre-MDT0000: new disk, initializing [ 3560.363663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.395310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3568.287136] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3581.623111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.768290] Lustre: 120247:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3581.857517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3581.860656] Lustre: Skipped 1 previous similar message [ 3581.917633] Lustre: lustre-MDT0001: new disk, initializing [ 3582.017241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3582.064365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3582.078275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3586.421549] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3594.108491] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3597.356578] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3604.304358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.732673] Lustre: lustre-OST0000: new disk, initializing [ 3604.742196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3604.752789] Lustre: 121992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3604.895952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.482146] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3611.752936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3611.774430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3611.958796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3620.603996] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.086211] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3626.414499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3632.429524] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3632.819922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3637.729710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3637.737877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.751043] LustreError: Skipped 1 previous similar message [ 3637.757107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3637.771892] Lustre: Skipped 2 previous similar messages [ 3637.818140] Lustre: Skipped 1 previous similar message [ 3642.092658] Lustre: server umount lustre-OST0000 complete [ 3647.968543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.988840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3648.004451] Lustre: Skipped 1 previous similar message [ 3653.220370] Lustre: server umount lustre-MDT0000 complete [ 3657.240632] LustreError: 120240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384147 with bad export cookie 436244792367191319 [ 3657.244929] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3657.261144] LustreError: 120240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3657.655765] Lustre: server umount lustre-MDT0001 complete [ 3666.806294] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 03:09:15 (1780384155) [ 3673.559827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.118984] LustreError: 124258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.139610] LustreError: 124258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3674.203365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.313933] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3679.202602] LustreError: 124259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3684.321083] LustreError: 124258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.004694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.125342] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3697.581459] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3701.416604] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3708.440912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.756624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.763946] Lustre: Skipped 1 previous similar message [ 3715.852333] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3722.107364] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3726.693918] Lustre: *** cfs_fail_loc=159, val=0*** [ 3743.102136] Lustre: lustre-MDT0000: Client 866487b5-8832-42a2-9236-9493c9c3fc9a (at 192.168.202.20@tcp) reconnecting [ 3744.786913] Lustre: *** cfs_fail_loc=159, val=0*** [ 3744.799160] Lustre: Skipped 5 previous similar messages [ 3761.006711] Lustre: lustre-MDT0000: Client 866487b5-8832-42a2-9236-9493c9c3fc9a (at 192.168.202.20@tcp) reconnecting [ 3779.553650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3779.571574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.590629] Lustre: Skipped 1 previous similar message [ 3779.597811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.583586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.678511] Lustre: server umount lustre-OST0000 complete [ 3789.792045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3789.798839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.805197] Lustre: Skipped 1 previous similar message [ 3789.815585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.826785] Lustre: Skipped 2 previous similar messages [ 3803.104960] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3803.327699] Lustre: server umount lustre-MDT0000 complete [ 3805.168418] LustreError: 126572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.194328] LustreError: 126572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3807.638204] LustreError: 124240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384297 with bad export cookie 436244792367192201 [ 3807.657544] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.658170] LustreError: 124240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3807.977342] Lustre: server umount lustre-MDT0001 complete [ 3820.139887] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 3822.804458] Key type lgssc unregistered [ 3823.104553] LNet: 127679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3823.110079] LNetError: 127679:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3824.166735] LNet: Removed LNI 192.168.202.120@tcp [ 3825.239636] Key type .llcrypt unregistered [ 3825.241741] Key type ._llcrypt unregistered [ 3850.264321] Key type ._llcrypt registered [ 3850.266539] Key type .llcrypt registered [ 3850.391707] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 3852.254626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3852.351864] alg: No test for adler32 (adler32-zlib) [ 3853.357626] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3853.681337] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 3855.368523] Key type lgssc registered [ 3856.708961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3865.645711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3865.681201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.235313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.661357] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3877.358536] LustreError: 129077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.385070] LustreError: 129077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3881.066792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.715079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3886.226980] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3892.055261] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3896.151538] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3903.809144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.257903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.485186] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 3912.472194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3915.565511] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3924.571996] Lustre: *** cfs_fail_loc=159, val=0*** [ 3941.201050] Lustre: lustre-MDT0000: Client 7dcce559-1568-41f2-8400-4154be482931 (at 192.168.202.20@tcp) reconnecting [ 3943.674705] Lustre: *** cfs_fail_loc=159, val=0*** [ 3960.185157] Lustre: lustre-MDT0001: Client 7dcce559-1568-41f2-8400-4154be482931 (at 192.168.202.20@tcp) reconnecting [ 3962.149867] Lustre: *** cfs_fail_loc=159, val=0*** [ 3962.151663] Lustre: Skipped 3 previous similar messages [ 3977.573627] Lustre: lustre-MDT0001: Client 7dcce559-1568-41f2-8400-4154be482931 (at 192.168.202.20@tcp) reconnecting [ 3982.875520] Lustre: 131775:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3985.539111] Lustre: *** cfs_fail_loc=159, val=0*** [ 3985.550018] Lustre: Skipped 4 previous similar messages [ 4002.149123] Lustre: lustre-MDT0000: Client fb838b9a-825f-4166-a66a-ff3723c52b8b (at 192.168.202.20@tcp) reconnecting [ 4003.810293] Lustre: *** cfs_fail_loc=159, val=0*** [ 4003.818265] Lustre: Skipped 13 previous similar messages [ 4019.497985] Lustre: lustre-MDT0000: Client fb838b9a-825f-4166-a66a-ff3723c52b8b (at 192.168.202.20@tcp) reconnecting [ 4021.013848] Lustre: 132067:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4040.674478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.685464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.686454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.697369] Lustre: Skipped 1 previous similar message [ 4040.715239] Lustre: Skipped 1 previous similar message [ 4045.796337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.804335] Lustre: Skipped 1 previous similar message [ 4046.581611] Lustre: server umount lustre-OST0000 complete [ 4050.923040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.936898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4050.941236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4050.944368] Lustre: Skipped 1 previous similar message [ 4056.036943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4056.047109] Lustre: Skipped 2 previous similar messages [ 4056.941163] Lustre: server umount lustre-MDT0000 complete [ 4060.267636] LustreError: 131896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384550 with bad export cookie 8355987653729262418 [ 4060.277840] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4060.286809] LustreError: 131896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4060.833323] Lustre: server umount lustre-MDT0001 complete [ 4072.145838] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4075.299059] Key type lgssc unregistered [ 4075.616380] LNet: 133133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4075.625587] LNetError: 133133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4075.645214] LNet: Removed LNI 192.168.202.120@tcp [ 4076.302309] Key type .llcrypt unregistered [ 4076.304253] Key type ._llcrypt unregistered [ 4102.152539] Key type ._llcrypt registered [ 4102.156407] Key type .llcrypt registered [ 4102.280573] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4103.403063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.530668] alg: No test for adler32 (adler32-zlib) [ 4104.562896] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4104.820368] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4106.631206] Key type lgssc registered [ 4107.881978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.832157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4115.847541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.453172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.277725] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4127.729972] LustreError: 134533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.753300] LustreError: 134533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4130.862988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.361092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4136.473572] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4142.636615] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4146.321324] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4152.972982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.253771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.341332] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4159.105445] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4162.481958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4163.754621] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4169.034205] Lustre: 136776:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4173.096665] Lustre: 136824:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4174.574258] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4175.559744] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4179.978869] Lustre: 136971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4181.019906] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4181.824210] Lustre: 137071:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4187.104744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4187.112623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.125105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.129769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.132077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.138412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4191.701933] Lustre: server umount lustre-OST0000 complete [ 4197.857219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4197.867562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.887697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4197.906074] Lustre: Skipped 1 previous similar message [ 4203.492768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4203.501136] Lustre: Skipped 3 previous similar messages [ 4208.614047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.635376] Lustre: Skipped 1 previous similar message [ 4210.143571] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4210.501454] Lustre: server umount lustre-MDT0000 complete [ 4213.729745] LustreError: 134529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.751212] LustreError: 134529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4214.836949] LustreError: 135251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384704 with bad export cookie 7883578312233198669 [ 4214.841313] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4214.847177] LustreError: 135251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4215.143344] Lustre: server umount lustre-MDT0001 complete [ 4226.137459] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4228.867847] Key type lgssc unregistered [ 4229.242493] LNet: 138115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4229.252872] LNetError: 138115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4230.309877] LNet: Removed LNI 192.168.202.120@tcp [ 4231.108529] Key type .llcrypt unregistered [ 4231.110640] Key type ._llcrypt unregistered [ 4254.263569] Key type ._llcrypt registered [ 4254.265587] Key type .llcrypt registered [ 4254.364884] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4255.556054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4255.605491] alg: No test for adler32 (adler32-zlib) [ 4256.788838] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4257.018413] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4258.687190] Key type lgssc registered [ 4259.874106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4267.524889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4267.539746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.902309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.562483] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4278.241163] LustreError: 139513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.263306] LustreError: 139513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4279.924308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.445124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.468574] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4289.570212] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4293.365320] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4299.724256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4300.144444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.268614] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4306.165110] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4312.296923] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4313.480367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4321.778324] Lustre: *** cfs_fail_loc=159, val=0*** [ 4337.999725] Lustre: lustre-MDT0000: Client f34e6b7f-9200-426c-a9a3-3cd2a2c31bee (at 192.168.202.20@tcp) reconnecting [ 4354.535549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.537421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4354.540381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4354.542603] Lustre: Skipped 1 previous similar message [ 4356.818669] Lustre: server umount lustre-OST0000 complete [ 4362.208806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4362.222404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.233879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.236994] Lustre: Skipped 1 previous similar message [ 4364.768500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4364.768541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.342658] Lustre: server umount lustre-MDT0000 complete [ 4369.892171] LustreError: 139513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.907902] LustreError: 139513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4370.053794] LustreError: 141785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384860 with bad export cookie 15453119462956832178 [ 4370.060253] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4370.071838] LustreError: 141785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4370.558505] Lustre: server umount lustre-MDT0001 complete [ 4379.236308] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4381.821979] Key type lgssc unregistered [ 4382.096024] LNet: 142819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4382.105072] LNetError: 142819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4383.145187] LNet: Removed LNI 192.168.202.120@tcp [ 4383.912721] Key type .llcrypt unregistered [ 4383.914888] Key type ._llcrypt unregistered [ 4405.703989] Key type ._llcrypt registered [ 4405.706436] Key type .llcrypt registered [ 4405.789763] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4406.785281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4406.825305] alg: No test for adler32 (adler32-zlib) [ 4407.805927] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4408.001001] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4409.647595] Key type lgssc registered [ 4410.515575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4416.875429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4416.893541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.265292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.639426] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4428.268046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4428.559142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4431.856376] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4436.307304] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4439.857130] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4445.669575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.970711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.480977] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4453.173917] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4455.993498] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4464.445322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4484.142748] Lustre: lustre-MDT0001: evicting ed277f28-e207-4f60-8e5c-e1eafb5aebcf (at 192.168.202.20@tcp) by administrative request [ 4484.150889] Lustre: Skipped 1 previous similar message [ 4514.788339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4514.797298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.812772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.811646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4515.814484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.818055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.833056] Lustre: Skipped 1 previous similar message [ 4517.783642] Lustre: server umount lustre-OST0000 complete [ 4520.930070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.935398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4520.955995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4520.969178] Lustre: Skipped 1 previous similar message [ 4526.048740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4526.055252] Lustre: Skipped 1 previous similar message [ 4526.898371] Lustre: server umount lustre-MDT0000 complete [ 4529.907738] LustreError: 144196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385019 with bad export cookie 16927559335446649230 [ 4529.912228] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4529.927245] LustreError: 144196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4530.219510] Lustre: server umount lustre-MDT0001 complete [ 4538.734181] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4541.050725] Key type lgssc unregistered [ 4541.374922] LNet: 147945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4541.389941] LNetError: 147945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4541.413170] LNet: Removed LNI 192.168.202.120@tcp [ 4542.254672] Key type .llcrypt unregistered [ 4542.259581] Key type ._llcrypt unregistered [ 4598.924987] Key type ._llcrypt registered [ 4598.927143] Key type .llcrypt registered [ 4599.025515] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4599.828370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4599.961936] alg: No test for adler32 (adler32-zlib) [ 4600.911659] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4601.068889] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4602.687345] Key type lgssc registered [ 4603.829466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4610.657853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4610.670408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.023973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.226893] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4622.314217] LustreError: 150416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.334513] LustreError: 150416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4623.646404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.126317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.224512] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4633.110726] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4636.376276] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4641.820926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.206277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.839343] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4649.395287] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4653.241673] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4657.590432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4664.802096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4664.807731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.833173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.874344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.875211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4667.875351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.880675] Lustre: Skipped 1 previous similar message [ 4670.330789] Lustre: server umount lustre-OST0000 complete [ 4675.553494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4675.561753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.574451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.114597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.122743] Lustre: Skipped 3 previous similar messages [ 4680.050683] Lustre: server umount lustre-MDT0000 complete [ 4683.244466] LustreError: 152572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.257014] LustreError: 152572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4683.535675] LustreError: 150394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385173 with bad export cookie 11192585247277434087 [ 4683.538288] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4683.544035] LustreError: 150394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4683.756446] Lustre: server umount lustre-MDT0001 complete [ 4692.015827] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4694.244370] Key type lgssc unregistered [ 4694.566025] LNet: 153600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4694.571522] LNetError: 153600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4694.590830] LNet: Removed LNI 192.168.202.120@tcp [ 4695.393579] Key type .llcrypt unregistered [ 4695.395718] Key type ._llcrypt unregistered [ 4715.071663] Key type ._llcrypt registered [ 4715.074113] Key type .llcrypt registered [ 4715.197602] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4716.245251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4716.267520] alg: No test for adler32 (adler32-zlib) [ 4717.308519] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4717.525694] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4719.223178] Key type lgssc registered [ 4720.227748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4732.597082] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 03:27:01 (1780385221) [ 4747.600386] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 4755.502178] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4782.971220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4782.990146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.159115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4784.179020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4784.243657] Lustre: lustre-MDT0000: new disk, initializing [ 4784.317689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.342985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4787.636309] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4795.869115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.967949] Lustre: 157800:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4796.042170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4796.045914] Lustre: Skipped 1 previous similar message [ 4796.086404] Lustre: lustre-MDT0001: new disk, initializing [ 4796.157849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4796.216153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4796.222230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4798.939198] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4804.187331] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4806.978481] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4811.380700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.564874] Lustre: lustre-OST0000: new disk, initializing [ 4811.568324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4811.577243] Lustre: 159538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4811.644055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.902491] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4821.532564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4821.545437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4821.688691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4821.716205] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4831.716546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.717547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.728433] Lustre: Skipped 1 previous similar message [ 4835.675862] Lustre: server umount lustre-OST0000 complete [ 4841.952821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.960988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4841.966974] Lustre: Skipped 1 previous similar message [ 4841.979615] Lustre: Skipped 2 previous similar messages [ 4844.656501] Lustre: server umount lustre-MDT0000 complete [ 4847.591481] LustreError: 157791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385337 with bad export cookie 11826700140359108229 [ 4847.601231] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4847.904562] Lustre: server umount lustre-MDT0001 complete [ 4856.158272] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 4857.925590] Key type lgssc unregistered [ 4858.289854] LNet: 161313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4858.296618] LNetError: 161313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4858.316705] LNet: Removed LNI 192.168.202.120@tcp [ 4859.012813] Key type .llcrypt unregistered [ 4859.014384] Key type ._llcrypt unregistered [ 4891.313589] Key type ._llcrypt registered [ 4891.315265] Key type .llcrypt registered [ 4891.543453] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 4893.837571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4893.890891] alg: No test for adler32 (adler32-zlib) [ 4895.106346] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4896.361379] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 4898.399246] Key type lgssc registered [ 4899.974276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4910.116918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4910.148247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.774580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.011699] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4921.321530] LustreError: 162712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.347315] LustreError: 162712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4925.447476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.823471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4930.515551] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4936.015879] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4939.103962] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4945.486220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4945.857120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.463171] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 4955.857989] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4956.079434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4962.332800] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4962.359809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.373900] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4962.381918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.394064] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4963.207560] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4963.236694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4963.251486] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4963.255392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.265887] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4964.032227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.769935] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4964.774900] Lustre: Skipped 1 previous similar message [ 4964.809956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.817276] Lustre: Skipped 1 previous similar message [ 4964.823223] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4964.827861] Lustre: Skipped 1 previous similar message [ 4964.833975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.849830] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4964.863270] Lustre: Skipped 1 previous similar message [ 4965.554930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.324160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.267441] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4967.272174] Lustre: Skipped 2 previous similar messages [ 4967.283099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.302659] Lustre: Skipped 2 previous similar messages [ 4967.305737] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4967.309814] Lustre: Skipped 2 previous similar messages [ 4967.313873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.326369] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4967.333046] Lustre: Skipped 2 previous similar messages [ 4968.138783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.871238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.532051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.394696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.226354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.985489] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4971.989829] Lustre: Skipped 5 previous similar messages [ 4972.007668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.015534] Lustre: Skipped 5 previous similar messages [ 4972.018294] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4972.021523] Lustre: Skipped 5 previous similar messages [ 4972.024236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.030973] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4972.041691] Lustre: Skipped 5 previous similar messages [ 4972.747435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.717470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.517595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.560193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.419871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.415365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.329084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.162139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.960375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.751583] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4980.758639] Lustre: Skipped 9 previous similar messages [ 4980.770520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.774856] Lustre: Skipped 9 previous similar messages [ 4980.776706] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4980.778391] Lustre: Skipped 9 previous similar messages [ 4980.780878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.787810] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4980.794592] Lustre: Skipped 9 previous similar messages [ 4981.520944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.449873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.448383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.371806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.261703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.156769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.882135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.750628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.536573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.398602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.343864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.317745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.354700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.255692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.284046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.089724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.027501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.955734] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4996.963434] Lustre: Skipped 17 previous similar messages [ 4996.972653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.982764] Lustre: Skipped 17 previous similar messages [ 4996.987963] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4996.992449] Lustre: Skipped 17 previous similar messages [ 4996.997758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.009910] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4997.012744] Lustre: Skipped 17 previous similar messages [ 4997.875508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.919879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.873887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.900174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.808540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.828596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.777101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.677298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.613407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.857576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.632349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.513810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.193460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.045762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.805630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.638727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.547689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.449609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.342596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.336821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.164646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.080532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.168272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.870733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.686736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.571205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.350139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.179842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.006411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.782438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.694762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.445425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.358373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.360293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.181979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.061715] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5029.066410] Lustre: Skipped 35 previous similar messages [ 5029.085938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.096979] Lustre: Skipped 35 previous similar messages [ 5029.102823] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5029.106643] Lustre: Skipped 35 previous similar messages [ 5029.110984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.116526] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5029.122279] Lustre: Skipped 35 previous similar messages [ 5030.081393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.033965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.964656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.066554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.106721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.167096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.238209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.430329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.458453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.511752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.830532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.739120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.533651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.672555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.674279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.701535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.721829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.689874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.567670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.657302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.504960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.640508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.458277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.659993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.522565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.484159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.350835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.356304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.138749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.871630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.863136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.603679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.353185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.227126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.013542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.856817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.878904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.887102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.825123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.851840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.797509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.661992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.734344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.490469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.486983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.256533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.112924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.975566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.104279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.055759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.155065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.051849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.077919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.940164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.923325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.832886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.877481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.744294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.935889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.911984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.023597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.862533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.741327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.676386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.681052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.859223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.766605] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5093.770075] Lustre: Skipped 66 previous similar messages [ 5093.782681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5093.789985] Lustre: Skipped 66 previous similar messages [ 5093.793624] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5093.796403] Lustre: Skipped 66 previous similar messages [ 5093.799993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.814867] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5093.828737] Lustre: Skipped 66 previous similar messages [ 5094.811564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.075866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.087468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.144362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.171208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.043202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.249645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.282485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.143582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.076624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.121846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.026328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.994830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.784073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.596416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.413065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.210081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.061622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.090395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.042972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.979090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.778412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.498991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.289474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.161882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.135766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.239871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.064736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.976562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.883172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.878228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.709265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.667728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.963331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.356883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.283386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.108514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.081463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.063490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.976918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.117864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.159559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.143658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.987602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.935588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.930868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.927188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.847628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.687692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.709204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.632112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.492248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.321795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.230930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.374098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.548731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.456011] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.549225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.516250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.765921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.614081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.566711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.502711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.552695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.737195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.734603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.979746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.995619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.113518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.298372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.492883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.620397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.612731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.664478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.631403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.505263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.345132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.430828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.280195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.267921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.273213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.223819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.102587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.983885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.848703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.841343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.742481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.779961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.657581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.570410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.416276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.401978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.188728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.035300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.111139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.051599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.872760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.620826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.355885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.287683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.122667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.180895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.996766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.906463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.886145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.994879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.015491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.939375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.972323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.810889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.848123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.818495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.679436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.575696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.432710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.243523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.147722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.028214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.971631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.018154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.045308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.252173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.197142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.265492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.180457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.206919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.288242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.148991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.939351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.736992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.652874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.403535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.382326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.167568] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5222.171434] Lustre: Skipped 133 previous similar messages [ 5222.181455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.189345] Lustre: Skipped 133 previous similar messages [ 5222.192726] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5222.196519] Lustre: Skipped 133 previous similar messages [ 5222.199512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.207642] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5222.210577] Lustre: Skipped 133 previous similar messages [ 5223.066245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.919455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.777530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.458501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.199985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.883875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.670310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.380261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.256942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.193405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.193272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.080116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.964231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.763646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.719617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.475631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.528789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.418440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.273603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.127993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.068657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.851183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.935058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.658864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.482134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.237090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.003569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.611405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.438323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.312696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.153946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.919323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.858150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.545396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.415536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.285572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.389635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.104863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.014190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.015566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.155026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.360262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.398528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.568563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.571514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.528088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.475444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.535436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.637629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.568501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.522309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.406306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.335236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.017525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.837653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.773399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.644867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.563735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.367615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.349102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.421656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.300142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.130064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.934351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.653937] LustreError: 162732:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5279.661778] LustreError: 162732:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5279.676095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.369143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.238516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.037069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.908426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.712184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.750166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.594380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.414213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.337677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.258486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.038470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.815420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.636481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.460075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.390863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.127682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.011401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.767511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.569892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.386179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.082217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.942163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.736362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.429907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.128231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.874835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.621900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.477229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.273075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.088466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.817603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.630795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.504440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.327765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.192504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.099381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.933947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.771389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.498186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.286889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.126985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.982291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.763549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.468470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.253224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.955027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.928493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.742911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.559159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.235566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.976160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.544791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.197934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.872993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.540122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.282355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.977136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.725234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.518844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.321785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.198322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.970451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.764525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.562482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.371561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.118609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.981174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.732963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.631277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.420028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.230775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.983983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.791919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.640593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.532450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.243406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.911427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.575371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.283527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.099841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.882243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.637383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.394036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.104280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.000953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.744793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.632516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.381119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.228603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.027389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.705118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.501913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.209502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.942507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.581363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.225168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.844997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.484922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.148960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.898917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.565185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.261553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.923526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.603049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.350089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.136043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.941862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.783717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.453242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.177062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.924694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.884974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.604688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.531131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.395246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.229429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.895620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.620872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.352942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.164552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.850699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.524919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.216669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.913174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.519380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.373783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.143501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.869330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.674783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.416611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.091108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.987067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.811345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.651910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.520715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.249122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.929324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.813973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.537654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.311272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.156277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.031804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.895537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.758386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.443082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.329521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.014824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.875634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.618579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.333964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.945444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.550611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.299640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.051466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.761494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.552389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.295325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.021654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.624824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.288458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.957803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.638208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.260825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.930827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.606073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.240908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.837914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.611751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.254608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.924234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.648136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.266922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.924579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.724287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.379111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.073973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.717770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.395098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.029970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.736117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.436605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.114763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.695676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.370442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.049115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.806046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.451595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.128944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.740444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.521174] LustreError: 162732:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5423.533613] LustreError: 162732:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5423.545144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.244070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.910515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.564110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.370223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.090654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.781363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.485096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.248614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.022624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.785284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.501645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.274614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.056536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.768875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.461607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.200983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.837890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.474574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.992343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.685832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.265076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.929587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.487525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.139826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.732150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.380504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.018642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.761291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.336911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.979584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.602503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.232138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.874894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.543482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.157186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.926120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.574270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.528288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.180237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.912269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.514895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.217350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.856446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.544850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.329352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.984467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.779716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.516340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.113432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.821915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.387664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.998791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.565171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.061316] LustreError: 162732:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5460.074873] LustreError: 162732:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5460.078673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.627902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.307142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.947399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.855539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.687472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.377636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.073901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.884891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.596266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.339569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.231634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.004564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.810554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.638652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.415192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.093864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.824310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.604328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.305791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.163551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.756853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.506165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.263393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.048612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.875539] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5478.885045] Lustre: Skipped 333 previous similar messages [ 5478.896180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5478.906832] Lustre: Skipped 333 previous similar messages [ 5478.910721] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5478.917147] Lustre: Skipped 333 previous similar messages [ 5478.921573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.933992] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5478.937308] Lustre: Skipped 333 previous similar messages [ 5479.667437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.464096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.326129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.084958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.881206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.645622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.484778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.368681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.099827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.881804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.746637] LustreError: 162732:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5487.758341] LustreError: 162732:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5487.765619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.447927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.251207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.106741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.881279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.573476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.416082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.230674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.057541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.764616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.630682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.347779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.097629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.900270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.706690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.485142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.240534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.222719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.107790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.830829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.642109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.493907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.199437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.741676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.520776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.277244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.028184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.753766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.476846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.135533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.973573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.692037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.408231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.098627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.775181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.470853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.223852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.875305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.551499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.258665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.930126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.461467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.156750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.993934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.894995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.546667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.421856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.015228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.586658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.245581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.994439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.773952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.551970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.341795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.114903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.958073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.708340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.461824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.230879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.003762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.631493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.366375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.163659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.899761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.536657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.139263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.753146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.563533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.325080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.037576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.846310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.664963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.389987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.212537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.951930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.689407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.552693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.469072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.336540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.258307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.001604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.806370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.501271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.160313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.840025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.606317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.426598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.274996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.094366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.916661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.656805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.346656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.083953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.971657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.696387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.384333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.249191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.142431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.047801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.807192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.697472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.481375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.271806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.171193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.040397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.929981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.854047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.488374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.402710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.231039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.971384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.625057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.491020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.202272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.140327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.895303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.625944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.314397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.132339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.080486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.003648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.623863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.174781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.846337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.575229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.217426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.915945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.722607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.625455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.413107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.319080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.087311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.999138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.776142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.438746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.244223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.049612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.846144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.719419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.559694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.521607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.363969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.078581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.855895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.535478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.364874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.103963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.932857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.723259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.608798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.412602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.235124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.027048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.759182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.664344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.559833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.369451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.097565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.162845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.952807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.853112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.690954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.587729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.470955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.295816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.156919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.834728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.585179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.400227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.176347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.840291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.660634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.411764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.212539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.850231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.685528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.410972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.162996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.951815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.850810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.539884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.377165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.055566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.893341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.584151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.491159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.563437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.496830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.324767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.128819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.922240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.967670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.883689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.806344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.596704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.484123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.294384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.152826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.072186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.872825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.520126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.333530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.022354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.743742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.464038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.347460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.157161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.950938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.679155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.417647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.211171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.003657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.722494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.436806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.024356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.735825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.463495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.316147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.969256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.718314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.411247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.340587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.101465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.904915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.557779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.249906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.860639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.428499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.194609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.033786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.670968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.390503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.229980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.079872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.741126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.403948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.027354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.652536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.285528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.042527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.673924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.322426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.971639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.573317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.283665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.040987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.783335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.485836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.314184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.120952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.744427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.374340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.029568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.849500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.587412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.261365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.837462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.400689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.098620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.770490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.318190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.006701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.527334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.187252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.838036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.422648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.085937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.602905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.130200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.755274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.402286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.996963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.531322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.204337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.838643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.318045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.836614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.359102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.902649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.517974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.135159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.665665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.135861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.603513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.121222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.628807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.122360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.648122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.167131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.655470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.149756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.614480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.073295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.527947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.025584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.557349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.114303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.664342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.194230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.715878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.220503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.817506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.426861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.123178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.778717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.368342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.038695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.734558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.458556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.177484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.968571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.736518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.518643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.156880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.830368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.580949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.307765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.052600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.851400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.633695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.384667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.091514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.843888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.460028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.238529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.900303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.671183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.498650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.264610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.952931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.585262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.169740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.789519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.419538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.069646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.720479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.413160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.026222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.678725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.365228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.051918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.741992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.405752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.093132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.672246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.496416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.150274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.792115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.344439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.882354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.473452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.210427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.845534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.530420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.179761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.931744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.662889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.446519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.111678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.924269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.672426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.407221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.101501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.925908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.520587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.257355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.018742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.720526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.521117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.337360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.137725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.826484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.633030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.389289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.194762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.047970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.751730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.362509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.644673] LustreError: 164861:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6037.651425] LustreError: 162732:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6288.884856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6288.888312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.899748] LustreError: Skipped 2 previous similar messages [ 6288.911056] Lustre: Skipped 388 previous similar messages [ 6288.917833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6293.668995] Lustre: server umount lustre-OST0000 complete [ 6298.079917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6298.085026] LustreError: Skipped 5 previous similar messages [ 6298.087914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.094763] Lustre: Skipped 2 previous similar messages [ 6299.616317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.623115] Lustre: Skipped 1 previous similar message [ 6304.737401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.741689] Lustre: Skipped 1 previous similar message [ 6309.857772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6309.860561] Lustre: Skipped 1 previous similar message [ 6310.368807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6310.540248] Lustre: server umount lustre-MDT0000 complete [ 6312.627682] LustreError: 207726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386802 with bad export cookie 10697953876422646578 [ 6312.632460] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.633518] LustreError: 207726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6312.857840] Lustre: server umount lustre-MDT0001 complete [ 6318.163562] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6319.520624] Key type lgssc unregistered [ 6319.688876] LNet: 217500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6319.692557] LNetError: 217500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6319.707160] LNet: Removed LNI 192.168.202.120@tcp [ 6320.115876] Key type .llcrypt unregistered [ 6320.117553] Key type ._llcrypt unregistered [ 6332.818883] Key type ._llcrypt registered [ 6332.820826] Key type .llcrypt registered [ 6332.875200] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6333.306963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6333.382436] alg: No test for adler32 (adler32-zlib) [ 6334.287807] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6334.411110] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6336.023163] Key type lgssc registered [ 6336.579237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6340.408284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6340.422520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.693365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.832515] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6348.493395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.702315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6350.724165] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6353.171758] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6354.902762] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6358.302646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.484536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.243651] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6364.002972] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6366.711320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6371.500434] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6372.295590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.302443] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6372.307301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.313068] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6372.313075] LustreError: 218896:0:(osp_precreate.c:1264:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6374.699739] Lustre: lustre-OST0000: Not available for connect from 192.168.202.20@tcp (stopping) [ 6374.880357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.882533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.953420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.955471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6377.965155] Lustre: Skipped 2 previous similar messages [ 6379.230931] Lustre: server umount lustre-OST0000 complete [ 6392.399109] LustreError: 218896:0:(osp_precreate.c:1264:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6408.034290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6408.202059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.446574] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6412.834481] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6415.723846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6417.575786] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9bdb47089000.ost_server_uuid 50 [ 6418.277642] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9bdb47089000.ost_server_uuid in FULL state after 0 sec [ 6418.470417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.478227] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6423.522618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.531982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.881179] Lustre: server umount lustre-OST0000 complete [ 6428.640704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6428.643083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6428.645376] Lustre: Skipped 1 previous similar message [ 6433.761288] LustreError: 219587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.768983] LustreError: 219587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6433.778951] Lustre: server umount lustre-MDT0000 complete [ 6435.535224] LustreError: 218856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386925 with bad export cookie 5135269096979925860 [ 6435.536279] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6435.540959] LustreError: 218856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6435.722643] Lustre: server umount lustre-MDT0001 complete [ 6440.490173] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6441.884965] Key type lgssc unregistered [ 6442.058537] LNet: 223514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6442.063123] LNetError: 223514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6442.081545] LNet: Removed LNI 192.168.202.120@tcp [ 6442.478162] Key type .llcrypt unregistered [ 6442.480093] Key type ._llcrypt unregistered [ 6457.022031] Key type ._llcrypt registered [ 6457.023743] Key type .llcrypt registered [ 6457.069711] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6457.540251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6457.551197] alg: No test for adler32 (adler32-zlib) [ 6458.436964] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6458.545283] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6460.143157] Key type lgssc registered [ 6460.609314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6465.038047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6465.047226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.243704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.778728] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6471.097594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.255675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6472.834879] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6474.007258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6476.615021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.758429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6477.802452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6479.088910] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6482.479685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.553225] Lustre: lustre-OST0001: new disk, initializing [ 6482.555128] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6482.559180] Lustre: 227463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6482.607786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6485.008385] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6489.089503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6489.092794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6489.128667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6490.961973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6496.864128] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6506.975847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6506.979451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.984595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.537555] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.538697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.543535] Lustre: Skipped 2 previous similar messages [ 6509.545719] Lustre: Skipped 2 previous similar messages [ 6511.852745] Lustre: server umount lustre-MDT0000 complete [ 6513.419135] LustreError: 227462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387003 with bad export cookie 1362331002697484845 [ 6513.419925] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.423043] LustreError: 227462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6513.548956] Lustre: server umount lustre-MDT0001 complete [ 6525.384763] Lustre: server umount lustre-OST0000 complete [ 6537.399759] Lustre: server umount lustre-OST0001 complete [ 6545.557827] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6549.452760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.642207] LustreError: 230525:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.650453] LustreError: 230525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6549.683819] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.212593] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6554.627295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.374778] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6557.580283] Lustre: 231630:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6560.584077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.725317] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.728182] Lustre: Skipped 1 previous similar message [ 6563.159829] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6564.837488] LustreError: 232505:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.123867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6569.705595] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6574.569422] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6574.569536] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 6578.631634] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6580.138080] Lustre: 233465:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6588.150719] Lustre: server umount mylustre-MDT0000 complete [ 6589.649625] LustreError: 233467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387079 with bad export cookie 1362331002697487512 [ 6589.655728] LustreError: 233467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6589.656147] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.921208] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.921894] LustreError: 230520:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.924082] Lustre: mylustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6589.927937] Lustre: Skipped 2 previous similar messages [ 6589.940097] LustreError: 230520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6591.800669] Lustre: server umount mylustre-MDT0001 complete [ 6593.431959] Lustre: server umount mylustre-OST0000 complete [ 6594.978744] Lustre: server umount mylustre-OST0001 complete [ 6603.247910] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6607.249190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.434775] LustreError: 235752:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.477144] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.480157] Lustre: Skipped 1 previous similar message [ 6608.967288] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6612.027364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.591987] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6614.600924] Lustre: 236860:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6616.920172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.086213] LustreError: 237215:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.092120] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 6619.164869] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6622.278591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.651156] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6627.822584] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6628.228896] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6629.676571] Lustre: 238695:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6632.928739] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6632.934087] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.937348] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.939469] Lustre: Skipped 3 previous similar messages [ 6632.945501] Lustre: Skipped 4 previous similar messages [ 6637.974529] Lustre: server umount tfs-MDT0000 complete [ 6638.049947] LustreError: 235748:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.056507] LustreError: 235748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6639.321249] LustreError: 235729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387129 with bad export cookie 1362331002697489402 [ 6639.326176] LustreError: 235729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6639.327156] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.433116] Lustre: server umount tfs-MDT0001 complete [ 6650.376232] Lustre: server umount tfs-OST0000 complete [ 6661.981871] Lustre: server umount tfs-OST0001 complete [ 6669.755709] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6673.556903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.752321] LustreError: 240923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.786566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6673.788856] Lustre: Skipped 3 previous similar messages [ 6675.297752] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6678.647196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.407833] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6681.525643] Lustre: 242031:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6684.040627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6686.475870] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6690.351847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6692.646687] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6698.985707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 6698.986042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6707.065289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6708.519450] Lustre: 243887:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6711.127745] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 04:00:01 (1780387201) [ 6724.577531] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6724.580662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6724.582544] Lustre: Skipped 3 previous similar messages [ 6724.587073] Lustre: Skipped 3 previous similar messages [ 6734.821074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6734.823978] Lustre: Skipped 7 previous similar messages [ 6737.375125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6737.441294] Lustre: server umount lustre-MDT0000 complete [ 6738.775336] LustreError: 243889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387228 with bad export cookie 1362331002697491061 [ 6738.779362] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.780216] LustreError: 243889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6739.936077] LustreError: 242359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.936590] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.947675] LustreError: 242359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6739.953672] Lustre: Skipped 2 previous similar messages [ 6753.247121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6753.251576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6753.254899] Lustre: Skipped 7 previous similar messages [ 6753.352551] Lustre: server umount lustre-MDT0001 complete [ 6769.119156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6769.163668] Lustre: server umount lustre-OST0000 complete [ 6776.746719] Lustre: server umount lustre-OST0001 complete [ 6779.263408] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 6783.043065] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6796.679749] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6796.679749] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6796.679749] [ 6796.695412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6796.806990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6796.827157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6796.870237] Lustre: lustre-MDT0000: new disk, initializing [ 6796.931476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6796.933960] Lustre: Skipped 3 previous similar messages [ 6796.943695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6798.452921] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6803.350971] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 6803.350971] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6803.350971] [ 6803.361515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6803.416905] Lustre: 247045:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6803.448810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6803.451521] Lustre: Skipped 1 previous similar message [ 6803.488944] Lustre: lustre-MDT0001: new disk, initializing [ 6803.543504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6803.547833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6805.150689] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6809.997497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.137677] Lustre: lustre-OST0000: new disk, initializing [ 6810.140124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6810.144181] Lustre: 248595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6811.342202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6811.347217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6811.404150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6812.669084] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6817.332642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.402990] Lustre: lustre-OST0001: new disk, initializing [ 6817.404936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6817.408064] Lustre: 249449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6818.834545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6818.838483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6818.878423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6819.688520] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6824.809977] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 04:01:54 (1780387314) [ 6828.956740] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 04:01:58 (1780387318) [ 6842.336378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.336462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.339268] LustreError: Skipped 1 previous similar message [ 6842.345072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.349298] Lustre: Skipped 1 previous similar message [ 6846.532790] Lustre: server umount lustre-OST0000 complete [ 6849.504876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6849.508171] LustreError: 248742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.515896] LustreError: 248742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6854.237360] Lustre: server umount lustre-MDT0000 complete [ 6855.730893] LustreError: 249448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387345 with bad export cookie 1362331002697492608 [ 6855.734599] LustreError: 249448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6855.734737] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6861.962913] Lustre: server umount lustre-MDT0001 complete [ 6866.144403] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6885.343150] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.434044] Lustre: server umount lustre-OST0001 complete [ 6887.989561] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 6895.016828] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6911.279836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.377114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6911.394753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6911.433929] Lustre: lustre-MDT0000: new disk, initializing [ 6911.471080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6913.139539] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6918.144201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.198091] Lustre: 253907:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6918.319339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6919.841850] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6923.417524] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6924.846280] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6927.713435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6927.854667] Lustre: lustre-OST0000: new disk, initializing [ 6927.855873] Lustre: Skipped 1 previous similar message [ 6927.857886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6927.862402] Lustre: Skipped 2 previous similar messages [ 6927.866871] Lustre: 255650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6927.908905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.914623] Lustre: Skipped 5 previous similar messages [ 6929.282531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6929.286514] Lustre: Skipped 1 previous similar message [ 6929.288050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6929.333339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6930.266789] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6933.667409] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6939.616810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6939.618135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.620592] LustreError: Skipped 1 previous similar message [ 6939.625753] Lustre: Skipped 5 previous similar messages [ 6939.626571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6939.632507] Lustre: Skipped 8 previous similar messages [ 6942.365281] Lustre: server umount lustre-OST0000 complete [ 6950.067267] Lustre: server umount lustre-MDT0000 complete [ 6951.617287] LustreError: 253899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387441 with bad export cookie 1362331002697494078 [ 6951.620223] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6951.622381] LustreError: 253899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6951.750216] Lustre: server umount lustre-MDT0001 complete [ 6955.637312] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 6956.787271] Key type lgssc unregistered [ 6956.924400] LNet: 257277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6956.927296] LNetError: 257277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6956.934427] LNet: Removed LNI 192.168.202.120@tcp [ 6957.260168] Key type .llcrypt unregistered [ 6957.261905] Key type ._llcrypt unregistered [ 6967.807751] Key type ._llcrypt registered [ 6967.809304] Key type .llcrypt registered [ 6967.852981] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 6968.287474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6968.296151] alg: No test for adler32 (adler32-zlib) [ 6969.164145] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6969.273862] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 6970.863198] Key type lgssc registered [ 6971.331021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6974.407983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6974.416970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.598053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.103726] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6980.396716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6980.551530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6982.022892] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6983.957318] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6985.344533] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6987.946849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.079635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.188182] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 6992.141069] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6992.518816] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 6995.425274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6995.428076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.432292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6999.394576] Lustre: server umount lustre-OST0000 complete [ 7001.055941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7001.059265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.065056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.664109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.664752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7010.784237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7010.786388] Lustre: Skipped 2 previous similar messages [ 7015.391235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7015.476276] Lustre: server umount lustre-MDT0000 complete [ 7015.903986] LustreError: 258697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.910678] LustreError: 258697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7016.910310] LustreError: 258679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387506 with bad export cookie 14061758704839199158 [ 7016.912431] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.915130] LustreError: 258679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7017.047499] Lustre: server umount lustre-MDT0001 complete [ 7020.933605] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7022.003992] Key type lgssc unregistered [ 7022.131354] LNet: 261922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7022.134285] LNetError: 261922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7022.141325] LNet: Removed LNI 192.168.202.120@tcp [ 7022.449466] Key type .llcrypt unregistered [ 7022.450675] Key type ._llcrypt unregistered [ 7029.715315] Key type ._llcrypt registered [ 7029.716861] Key type .llcrypt registered [ 7029.759867] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7030.083271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7030.108287] alg: No test for adler32 (adler32-zlib) [ 7030.966105] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7031.059913] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7032.647163] Key type lgssc registered [ 7033.081858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7036.568286] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7036.574249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7051.999335] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7057.119176] LustreError: 263077:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.120@tcp: failed processing log, type 4: rc = -110 [ 7083.743194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.627318] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7087.442062] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7088.188288] Lustre: server umount lustre-OST0000 complete [ 7093.567905] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7094.668647] Key type lgssc unregistered [ 7094.802474] LNet: 264574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7094.805665] LNetError: 264574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7094.816322] LNet: Removed LNI 192.168.202.120@tcp [ 7095.123510] Key type .llcrypt unregistered [ 7095.124922] Key type ._llcrypt unregistered [ 7109.518210] Key type ._llcrypt registered [ 7109.519450] Key type .llcrypt registered [ 7109.560987] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7109.899732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7109.954055] alg: No test for adler32 (adler32-zlib) [ 7110.826818] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7110.912417] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7112.496681] Key type lgssc registered [ 7112.936818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7117.844755] loop0: detected capacity change from 0 to 204800000 [ 7117.955258] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7117.960615] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7118.094971] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7118.518607] loop0: detected capacity change from 0 to 204800000 [ 7118.593055] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7118.598536] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7118.745490] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7119.146624] loop0: detected capacity change from 0 to 204800000 [ 7119.229777] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7119.236492] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7119.286083] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7119.720833] loop0: detected capacity change from 0 to 204800000 [ 7119.790832] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7120.281392] loop0: detected capacity change from 0 to 204800000 [ 7120.297738] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7121.798649] loop0: detected capacity change from 0 to 204800000 [ 7121.809059] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7123.246458] loop0: detected capacity change from 0 to 204800000 [ 7123.258925] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7124.637058] loop0: detected capacity change from 0 to 204800000 [ 7124.646719] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7126.068218] loop0: detected capacity change from 0 to 204800000 [ 7126.094683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7126.104983] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7126.142163] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7126.145418] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 7126.148573] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 7126.152177] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 7126.156193] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7126.160923] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7126.165323] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7126.169459] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 7126.173498] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 7126.178675] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 7126.183029] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 7126.186939] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 7126.191036] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 7126.194975] Lustre: 267943:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 7128.911113] Lustre: server umount MGS complete [ 7129.218517] loop0: detected capacity change from 0 to 204800000 [ 7129.247600] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.423047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7129.444414] Lustre: lustre-MDT0000: Aborting client recovery [ 7129.776790] loop1: detected capacity change from 0 to 204800000 [ 7129.813349] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.849191] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7129.855163] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 7129.860980] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 7129.866604] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7129.871425] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7129.876184] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7129.881180] Lustre: 268375:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 7129.888823] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7129.970727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7129.995437] Lustre: lustre-MDT0001: Aborting client recovery [ 7130.328256] loop2: detected capacity change from 0 to 204800000 [ 7130.393789] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.423611] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7130.428845] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7130.434133] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7130.441161] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7130.445834] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7130.449917] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7130.454930] Lustre: 268455:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7130.508200] Lustre: Found index 0 for lustre-OST0000, updating log [ 7130.521850] Lustre: 268473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7130.550138] Lustre: lustre-OST0000: Aborting recovery [ 7130.874897] loop3: detected capacity change from 0 to 204800000 [ 7130.905245] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.938241] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7130.943292] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7130.948087] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7130.953440] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7130.957599] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7130.963144] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7130.968920] Lustre: 268539:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7131.014054] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7131.016239] Lustre: Skipped 1 previous similar message [ 7136.232556] LustreError: 268319:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 7139.298810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7139.299678] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 7139.299859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7139.300778] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 7139.310278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 7139.310293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 7141.645461] LustreError: 268286:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 291us: rc = -2 [ 7154.658988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.660528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7154.664178] Lustre: Skipped 1 previous similar message [ 7159.080863] Lustre: server umount lustre-MDT0000 complete [ 7159.406694] LustreError: 268381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387649 with bad export cookie 2526390494169232432 [ 7159.408344] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7159.411788] LustreError: 268381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7159.775684] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.775869] LustreError: 268284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.779987] Lustre: Skipped 2 previous similar messages [ 7159.785067] LustreError: 268284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7159.787160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7159.791320] Lustre: Skipped 4 previous similar messages [ 7164.895473] LustreError: 268285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.895697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7164.900505] LustreError: 268285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7164.902605] Lustre: Skipped 1 previous similar message [ 7165.633664] Lustre: server umount lustre-MDT0001 complete [ 7180.255095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7180.295985] Lustre: server umount lustre-OST0000 complete [ 7186.716445] Lustre: server umount lustre-OST0001 complete [ 7189.491857] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 04:07:59 (1780387679) [ 7198.769820] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7202.115193] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7214.554592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.647976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7214.660530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7214.699399] Lustre: lustre-MDT0000: new disk, initializing [ 7214.731495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.739153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7216.113643] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7220.321375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.365804] Lustre: 272576:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7220.394603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7220.396887] Lustre: Skipped 1 previous similar message [ 7220.431157] Lustre: lustre-MDT0001: new disk, initializing [ 7220.468683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7220.481464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7220.485033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7221.826000] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7224.945143] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7226.111401] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7228.300596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7228.397825] Lustre: lustre-OST0000: new disk, initializing [ 7228.400133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7228.403540] Lustre: 274318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7228.406250] Lustre: 274318:0:(osd_compat.c:1352:osd_object_spec_find()) Skipped 1 previous similar message [ 7230.217526] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7233.255736] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7233.525857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7233.530244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7233.556994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7240.627203] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7250.827192] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7258.029674] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7294.945817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7294.945923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.948921] LustreError: Skipped 1 previous similar message [ 7294.954416] Lustre: Skipped 1 previous similar message [ 7294.956839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7294.959353] Lustre: Skipped 1 previous similar message [ 7300.065711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7300.068791] Lustre: Skipped 1 previous similar message [ 7300.627467] Lustre: server umount lustre-OST0000 complete [ 7302.623869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7302.626522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.248738] Lustre: server umount lustre-MDT0000 complete [ 7309.462399] LustreError: 273354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387799 with bad export cookie 2526390494169235134 [ 7309.464842] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7309.466508] LustreError: 273354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7309.580247] Lustre: server umount lustre-MDT0001 complete [ 7312.407027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.716318] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7315.312793] Lustre: server umount MGS complete [ 7319.350734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.507552] LustreError: 278334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.515414] LustreError: 278347:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7319.540656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7319.542790] Lustre: Skipped 1 previous similar message [ 7320.889904] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7323.806021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.239653] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7326.951704] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7328.156940] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7330.432438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.400289] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7334.215649] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7359.455848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7359.459212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.463435] Lustre: Skipped 1 previous similar message [ 7359.465688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.467932] Lustre: Skipped 3 previous similar messages [ 7364.881151] Lustre: server umount lustre-OST0000 complete [ 7370.207863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7370.211174] LustreError: Skipped 1 previous similar message [ 7370.212596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.216134] Lustre: Skipped 1 previous similar message [ 7372.369688] Lustre: server umount lustre-MDT0000 complete [ 7373.648359] LustreError: 278316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387863 with bad export cookie 2526390494169236961 [ 7373.651773] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7373.652520] LustreError: 278316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7373.765319] Lustre: server umount lustre-MDT0001 complete [ 7377.340465] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7378.394732] Key type lgssc unregistered [ 7378.519411] LNet: 281940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7378.522166] LNetError: 281940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7378.529307] LNet: Removed LNI 192.168.202.120@tcp [ 7378.828326] Key type .llcrypt unregistered [ 7378.829318] Key type ._llcrypt unregistered [ 7391.302846] Key type ._llcrypt registered [ 7391.304342] Key type .llcrypt registered [ 7391.341745] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7397.480620] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7397.822544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7397.867789] alg: No test for adler32 (adler32-zlib) [ 7398.729678] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7398.816364] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7400.399140] Key type lgssc registered [ 7400.796833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7413.935772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7413.946633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.049717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7415.061377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7415.098195] Lustre: lustre-MDT0000: new disk, initializing [ 7415.123136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.130089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7416.505818] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7421.108170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.156986] Lustre: 285834:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7421.184586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7421.187428] Lustre: Skipped 1 previous similar message [ 7421.225188] Lustre: lustre-MDT0001: new disk, initializing [ 7421.277657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7421.293247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7421.298156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7422.753236] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7426.021376] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7427.295863] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7429.772996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.874185] Lustre: lustre-OST0000: new disk, initializing [ 7429.876338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7429.880121] Lustre: 287574:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7429.908736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.910614] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7435.247823] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7436.793906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7436.798296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7436.832765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7445.057741] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7455.583362] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7460.218714] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7465.827882] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 7508.449763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7508.450168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.453104] LustreError: Skipped 1 previous similar message [ 7508.457852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7508.461087] Lustre: Skipped 1 previous similar message [ 7509.712522] Lustre: server umount lustre-OST0000 complete [ 7513.568450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7513.568636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.569539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7513.576709] Lustre: Skipped 2 previous similar messages [ 7518.688311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7518.690445] Lustre: Skipped 2 previous similar messages [ 7523.808097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.810547] Lustre: Skipped 1 previous similar message [ 7525.343264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7525.412619] Lustre: server umount lustre-MDT0000 complete [ 7526.652091] LustreError: 285826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388016 with bad export cookie 13037332716343508380 [ 7526.655919] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.656392] LustreError: 285826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7526.773040] Lustre: server umount lustre-MDT0001 complete [ 7529.521821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.762620] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7532.309647] Lustre: server umount MGS complete [ 7535.897803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.051034] LustreError: 291509:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7536.072354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7537.456590] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7540.280974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.409188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7541.705636] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7543.492611] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7544.640572] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7546.874459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.757818] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7550.482287] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7586.271863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7586.274122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.279318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.894294] Lustre: server umount lustre-OST0000 complete [ 7591.903824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7591.906425] LustreError: Skipped 1 previous similar message [ 7591.907457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.911577] Lustre: Skipped 1 previous similar message [ 7597.203898] Lustre: server umount lustre-MDT0000 complete [ 7597.536894] LustreError: 291496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7597.541760] LustreError: 291496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7598.498628] LustreError: 292217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388088 with bad export cookie 13037332716343510221 [ 7598.502441] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7598.503974] LustreError: 292217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7598.611040] Lustre: server umount lustre-MDT0001 complete [ 7601.948649] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7602.909469] Key type lgssc unregistered [ 7603.023415] LNet: 295105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7603.025695] LNetError: 295105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7603.033288] LNet: Removed LNI 192.168.202.120@tcp [ 7603.305757] Key type .llcrypt unregistered [ 7603.307238] Key type ._llcrypt unregistered [ 7609.576068] Key type ._llcrypt registered [ 7609.576884] Key type .llcrypt registered [ 7609.608668] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7615.207044] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7615.538287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7615.576032] alg: No test for adler32 (adler32-zlib) [ 7616.423442] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7616.502861] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7618.079120] Key type lgssc registered [ 7618.429393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7625.772133] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7630.637538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7630.644090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.731112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7631.742638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7631.778602] Lustre: lustre-MDT0000: new disk, initializing [ 7631.805867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.813271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7633.044280] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7637.004176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.094445] Lustre: lustre-OST0000: new disk, initializing [ 7637.095985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7637.097522] Lustre: Skipped 1 previous similar message [ 7637.099220] Lustre: 299059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7637.120797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7637.482300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7637.485886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7637.493613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7638.909252] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7826.912271] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.915875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.012309] Lustre: server umount lustre-MDT0000 complete [ 7831.269417] LustreError: 298263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388321 with bad export cookie 9983138875442239756 [ 7831.271451] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7831.274485] LustreError: 298263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7831.344503] Lustre: server umount lustre-OST0000 complete [ 7839.736956] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 7842.876152] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7854.920697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.020947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7855.039949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7855.081404] Lustre: lustre-MDT0000: new disk, initializing [ 7855.118724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.127982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7856.481490] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7860.607364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.648163] Lustre: 303236:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7860.672242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7860.673975] Lustre: Skipped 1 previous similar message [ 7860.707658] Lustre: lustre-MDT0001: new disk, initializing [ 7860.737995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7860.748866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7860.752350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7862.079141] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7865.194306] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7866.454430] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7868.906473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7869.006748] Lustre: lustre-OST0000: new disk, initializing [ 7869.009273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7869.012741] Lustre: 304975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7869.035167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.933243] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7873.525907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7873.528810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7873.561414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7874.097900] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7875.491238] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7875.560148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7876.855892] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7876.950754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7878.625202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.626155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7878.627194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7878.629089] Lustre: Skipped 1 previous similar message [ 7878.631905] LustreError: Skipped 1 previous similar message [ 7879.632605] Lustre: server umount lustre-OST0000 complete [ 7881.183782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7881.186205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.190373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.192316] Lustre: Skipped 1 previous similar message [ 7883.744222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.744780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.928358] Lustre: server umount lustre-MDT0000 complete [ 7888.225206] LustreError: 303226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388378 with bad export cookie 9983138875445180533 [ 7888.228245] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.229177] LustreError: 303226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7888.343559] Lustre: server umount lustre-MDT0001 complete [ 7889.534292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 7890.093834] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 04:19:40 (1780388380) [ 7892.278107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.448469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7893.771862] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7896.677581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.087153] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7899.779906] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7900.946972] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7903.115356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7903.201258] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7903.204173] Lustre: Skipped 1 previous similar message [ 7905.352821] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7907.265592] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7909.743152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7909.813557] Lustre: lustre-OST0001: new disk, initializing [ 7909.815377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7909.817552] Lustre: 309633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7909.849096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7909.852027] Lustre: Skipped 2 previous similar messages [ 7910.794472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7910.798634] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7910.819288] LustreError: 309779:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7910.823367] LustreError: 309779:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 7911.808199] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7911.843230] LustreError: 309790:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7911.847134] LustreError: 309790:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 7912.866692] LustreError: 309779:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7912.870345] LustreError: 309779:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 7914.915870] LustreError: 309790:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7914.919398] LustreError: 309790:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 7915.601207] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7915.676958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7916.939783] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9bdb50a26000.ost_server_uuid 50 [ 7917.434100] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9bdb50a26000.ost_server_uuid in FULL state after 0 sec [ 7918.662349] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9bdb50a26000.ost_server_uuid 50 [ 7919.012320] LustreError: 309779:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7919.016842] LustreError: 309779:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 7919.200772] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9bdb50a26000.ost_server_uuid in FULL state after 0 sec [ 7920.039262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7931.364107] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.365121] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7931.369034] Lustre: Skipped 1 previous similar message [ 7945.183130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7945.224359] Lustre: server umount lustre-OST0001 complete [ 7946.722845] LustreError: 309197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.729531] LustreError: 309197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7947.232082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7947.232148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.234591] LustreError: Skipped 1 previous similar message [ 7951.843379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7951.844296] LustreError: 308731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.845881] Lustre: Skipped 11 previous similar messages [ 7951.852482] LustreError: 308731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7952.976489] Lustre: server umount lustre-OST0000 complete [ 7960.469387] Lustre: server umount lustre-MDT0000 complete [ 7961.729426] LustreError: 307120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388451 with bad export cookie 9983138875445181401 [ 7961.732547] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7961.733651] LustreError: 307120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7961.855767] Lustre: server umount lustre-MDT0001 complete [ 7965.252661] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 7966.277605] Key type lgssc unregistered [ 7966.414351] LNet: 312073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7966.417712] LNetError: 312073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7966.425314] LNet: Removed LNI 192.168.202.120@tcp [ 7966.726167] Key type .llcrypt unregistered [ 7966.727460] Key type ._llcrypt unregistered [ 7976.065369] Key type ._llcrypt registered [ 7976.067019] Key type .llcrypt registered [ 7976.102471] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 7976.407966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7976.423052] alg: No test for adler32 (adler32-zlib) [ 7977.269249] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7977.348102] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 7978.927143] Key type lgssc registered [ 7979.258186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7981.734784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7981.741345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.894647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.066954] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7985.704991] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7987.871783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.999130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.279188] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7992.443059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.549742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.254221] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 7995.865680] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7998.013993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.089884] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.828240] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8001.511919] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8007.336651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8007.337396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 8013.618452] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8013.690066] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8089.573785] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8089.574705] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.576122] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8089.582034] Lustre: Skipped 1 previous similar message [ 8094.692629] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8094.694344] Lustre: Skipped 2 previous similar messages [ 8099.811098] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8099.812993] Lustre: Skipped 1 previous similar message [ 8103.391115] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8103.422656] Lustre: server umount lustre-OST0001 complete [ 8104.930371] LustreError: 315000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.937201] LustreError: 315000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8106.976162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8106.979212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.984875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8110.050343] LustreError: 316069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.053274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8110.056029] LustreError: 316069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8110.062193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8115.169397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8115.170151] LustreError: 316437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8115.171024] Lustre: Skipped 3 previous similar messages [ 8115.176850] LustreError: 316437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8119.263156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8119.309045] Lustre: server umount lustre-OST0000 complete [ 8121.311779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8121.314029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.408097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8125.408641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8125.409913] Lustre: Skipped 1 previous similar message [ 8126.616767] Lustre: server umount lustre-MDT0000 complete [ 8127.885228] LustreError: 316780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388617 with bad export cookie 9935518629764342708 [ 8127.887345] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8127.892678] LustreError: 316780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8128.015153] Lustre: server umount lustre-MDT0001 complete [ 8131.485900] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8132.527495] Key type lgssc unregistered [ 8132.687424] LNet: 318018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8132.690340] LNetError: 318018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8132.704329] LNet: Removed LNI 192.168.202.120@tcp [ 8133.012962] Key type .llcrypt unregistered [ 8133.014156] Key type ._llcrypt unregistered [ 8150.786668] Key type ._llcrypt registered [ 8150.787604] Key type .llcrypt registered [ 8150.823766] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8151.192602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.223071] alg: No test for adler32 (adler32-zlib) [ 8152.066951] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8152.143773] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8153.719132] Key type lgssc registered [ 8154.049932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8156.859681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8156.866752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8158.023473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.322194] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8162.161228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.292962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8163.573090] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8164.491372] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8166.620646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.740297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8168.528481] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8168.803575] LustreError: 321719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.807443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 8168.810075] LustreError: 321719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8171.103273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.171424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8172.202585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 8172.998505] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8177.004478] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8180.454767] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8187.874770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.875210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8187.876713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.877845] Lustre: Skipped 2 previous similar messages [ 8189.781733] Lustre: server umount lustre-MDT0000 complete [ 8190.973656] LustreError: 320246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388681 with bad export cookie 7306945683597086478 [ 8190.976497] LustreError: 320246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8190.976740] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8191.086483] Lustre: server umount lustre-MDT0001 complete [ 8202.179517] Lustre: server umount lustre-OST0000 complete [ 8213.720779] Lustre: server umount lustre-OST0001 complete [ 8218.670548] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8221.768965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.890992] LustreError: 325431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.907715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.097134] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8225.699104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8227.020146] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8227.934925] Lustre: 326538:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8229.976547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8230.087109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.090255] Lustre: Skipped 1 previous similar message [ 8231.140253] LustreError: 327086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.142262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 8231.818618] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8234.553982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8235.689542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 8236.376550] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8239.286169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8240.336297] Lustre: 328373:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8246.240746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.241059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.244599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.247434] Lustre: Skipped 2 previous similar messages [ 8246.248952] Lustre: Skipped 4 previous similar messages [ 8247.316855] Lustre: server umount lustre-MDT0000 complete [ 8248.534260] LustreError: 325413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388738 with bad export cookie 7306945683597088095 [ 8248.537519] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.543407] LustreError: 325413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8248.657159] Lustre: server umount lustre-MDT0001 complete [ 8260.615465] Lustre: server umount lustre-OST0000 complete [ 8271.068829] Lustre: server umount lustre-OST0001 complete [ 8272.304369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 8272.840512] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 8273.391107] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 04:26:03 (1780388763) [ 8280.932092] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8284.026804] loop0: detected capacity change from 0 to 1099511627776 [ 8284.169267] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 8284.172160] *** VALIDATE xfs *** [ 8284.176492] XFS (loop0): Mounting V5 Filesystem [ 8284.186064] XFS (loop0): Ending clean mount [ 8284.598619] loop1: detected capacity change from 0 to 18691697672192 [ 8304.095547] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 8306.960303] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8317.832691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.913628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8317.924219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8317.956272] Lustre: lustre-MDT0000: new disk, initializing [ 8317.979330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8317.982389] Lustre: Skipped 1 previous similar message [ 8317.987502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8319.259594] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8323.265335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.303267] Lustre: 333432:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8323.327866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8323.330814] Lustre: Skipped 1 previous similar message [ 8323.371617] Lustre: lustre-MDT0001: new disk, initializing [ 8323.410535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8323.413877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8324.635566] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8327.587573] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8328.716538] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8330.891272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.985683] Lustre: lustre-OST0000: new disk, initializing [ 8330.987166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8330.989167] Lustre: 335171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8332.088184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8332.092432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8332.124646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8332.736963] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8335.695047] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8336.943386] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8337.007934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8338.199850] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8338.261827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8342.496987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.498899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.500972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8342.506144] Lustre: Skipped 2 previous similar messages [ 8342.508940] Lustre: Skipped 2 previous similar messages [ 8347.618290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8347.620014] Lustre: Skipped 1 previous similar message [ 8352.739109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8352.740732] Lustre: Skipped 1 previous similar message [ 8353.247107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8353.299516] Lustre: server umount lustre-OST0000 complete [ 8357.856492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.601554] Lustre: server umount lustre-MDT0000 complete [ 8361.822319] LustreError: 333423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388851 with bad export cookie 7306945683597089628 [ 8361.825323] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.825414] LustreError: 333423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8361.939100] Lustre: server umount lustre-MDT0001 complete [ 8364.513151] XFS (loop0): Unmounting Filesystem [ 8366.040057] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 04:27:36 (1780388856) [ 8368.132092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.271357] LustreError: 337580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.293509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.296652] Lustre: Skipped 2 previous similar messages [ 8369.481549] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8372.089918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.434334] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8375.104478] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8376.276387] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8378.376683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8380.200057] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8381.850768] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8403.423887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8403.425899] LustreError: Skipped 1 previous similar message [ 8403.427086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.430176] Lustre: Skipped 1 previous similar message [ 8403.431710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8403.433932] Lustre: Skipped 2 previous similar messages [ 8406.538735] Lustre: server umount lustre-OST0000 complete [ 8408.031815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8408.033933] LustreError: Skipped 1 previous similar message [ 8413.787925] Lustre: server umount lustre-MDT0000 complete [ 8414.688526] LustreError: 337581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.694800] LustreError: 337581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8415.134622] LustreError: 339849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388905 with bad export cookie 7306945683597090510 [ 8415.138205] LustreError: 339849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8415.139534] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8415.248916] Lustre: server umount lustre-MDT0001 complete [ 8418.666899] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 8419.673976] Key type lgssc unregistered [ 8419.818537] LNet: 340870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8419.821756] LNetError: 340870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8419.833324] LNet: Removed LNI 192.168.202.120@tcp [ 8420.262452] Key type .llcrypt unregistered [ 8420.264042] Key type ._llcrypt unregistered [ 8430.407781] Key type ._llcrypt registered [ 8430.409031] Key type .llcrypt registered [ 8430.444948] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 8430.893710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8430.900684] alg: No test for adler32 (adler32-zlib) [ 8431.752624] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8431.832886] LNet: Added LNI 192.168.202.120@tcp [8/256/0/180] [ 8433.407118] Key type lgssc registered [ 8433.730772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8436.106094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8436.112810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.250673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.445555] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8441.233834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.353790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8442.577421] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8444.214102] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8445.324104] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8447.381388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8447.456281] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8447.480761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8449.200226] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8450.984387] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8461.794603] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8461.794631] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.795445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8461.806371] Lustre: Skipped 1 previous similar message [ 8461.969411] Lustre: server umount lustre-MDT0001 complete [ 8462.629876] LustreError: 342390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.637692] LustreError: 342390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8465.501375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8465.533832] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8465.535681] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8465.548409] Lustre: 342384:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8465.652578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.012607] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8471.013806] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8475.108765] LustreError: 345853:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8475.111300] LustreError: 345853:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8475.114079] LustreError: 342387:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8475.132398] LustreError: 345856:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -17 [ 8475.135719] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 8475.135719] [ 8769.827898] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.20@tcp (stopping) [ 8769.830673] Lustre: Skipped 1 previous similar message [ 8773.088798] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8773.088917] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.094558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8773.095959] Lustre: Skipped 1 previous similar message [ 8775.442062] Lustre: server umount lustre-MDT0001 complete [ 8778.209753] LustreError: 342395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8778.216698] LustreError: 342395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8778.752441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.785987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8778.787524] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8778.800678] Lustre: 342385:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8778.899147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.101726] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 8784.358351] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8788.454080] LustreError: 358612:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8788.456744] LustreError: 358612:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8788.459187] LustreError: 342387:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8788.462055] LustreError: 342387:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8788.480630] LustreError: 358615:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [ 8788.483926] LustreError: 358615:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [ 8788.487761] LustreError: 358615:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -17 [ 8788.491856] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.120@tcp [ 8788.491856] [ 9086.434893] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.438474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9086.438497] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9086.443801] Lustre: Skipped 2 previous similar messages [ 9086.452860] Lustre: Skipped 2 previous similar messages [ 9096.674905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9096.677885] Lustre: Skipped 5 previous similar messages [ 9098.207151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9098.320974] Lustre: server umount lustre-MDT0001 complete [ 9101.792568] LustreError: 349327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9101.799308] LustreError: 349327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9103.834620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.905984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9103.909851] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9103.935975] Lustre: 342384:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9104.137144] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 9106.258957] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9109.486744] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9109.994059] LustreError: 369787:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9109.997643] LustreError: 369787:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9110.001762] LustreError: 342387:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9110.010726] LustreError: 342387:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9110.058601] LustreError: 369802:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 9110.069054] LustreError: 369802:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.120@tcp: cfg command failed: rc = -114 [ 9110.074258] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 9110.074258] [ 9421.792402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9421.794844] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.797067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.797073] Lustre: Skipped 1 previous similar message [ 9421.808229] Lustre: Skipped 2 previous similar messages [ 9426.869346] Lustre: server umount lustre-MDT0000 complete [ 9426.914323] LustreError: 342390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9426.919570] LustreError: 342390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9428.851874] LustreError: 343112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389918 with bad export cookie 2799670683229505889 [ 9428.853817] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9428.855468] LustreError: 343112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9432.032221] LustreError: 342391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.033136] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9443.295133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9443.299244] LustreError: 344541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.342307] LustreError: 344541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9443.657865] Lustre: server umount lustre-MDT0001 complete [ 9447.324294] Lustre: server umount lustre-OST0000 complete [ 9457.350914] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9470.627960] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9510.389169] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 04:46:39 (1780389999) [ 9516.838813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.207084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9517.247661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9517.402363] Lustre: lustre-MDT0000: new disk, initializing [ 9517.525306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9517.554786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9522.153589] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9532.422775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.504185] Lustre: 383357:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9532.548719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9532.554319] Lustre: Skipped 1 previous similar message [ 9532.603146] Lustre: lustre-MDT0001: new disk, initializing [ 9532.684451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9532.732443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9532.745865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9536.859498] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9543.304920] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9546.186753] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9551.704215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9552.007278] Lustre: lustre-OST0000: new disk, initializing [ 9552.015420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9552.025122] Lustre: 385100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9552.112889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9556.771154] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_default_debug -1 all [ 9558.080925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9558.090721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9558.172107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9562.455922] Lustre: DEBUG MARKER: oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9568.738356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.748433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9568.762251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.762260] Lustre: Skipped 6 previous similar messages [ 9574.502142] Lustre: server umount lustre-MDT0000 complete [ 9577.186424] LustreError: 383351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390067 with bad export cookie 2799670683229507828 [ 9577.190749] LustreError: MGC192.168.202.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.197566] LustreError: 383351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9577.410210] Lustre: server umount lustre-MDT0001 complete [ 9590.541794] Lustre: server umount lustre-OST0000 complete [ 9612.275928] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing set_hostid [ 9620.262451] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing load_modules_local [ 9643.462726] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9416 sec ========== 04:48:52 (1780390132) [ 9645.240342] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:48:54 (1780390134) === [ 9647.467937] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:48:57 (1780390137) === [ 9676.202423] Lustre: DEBUG MARKER: oleg220-server.virtnet: executing unload_modules_local [ 9678.303833] Key type lgssc unregistered [ 9678.533392] LNet: 391293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9678.539637] LNetError: 391293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9679.592762] LNet: Removed LNI 192.168.202.120@tcp [ 9680.188340] Key type .llcrypt unregistered [ 9680.190290] Key type ._llcrypt unregistered