[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506481022 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2891144K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003329] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007887] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009027] pid_max: default: 32768 minimum: 301 [ 0.010166] LSM: Security Framework initializing [ 0.012040] Yama: becoming mindful. [ 0.013049] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.023569] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028565] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032122] *** VALIDATE tmpfs *** [ 0.033478] *** VALIDATE proc *** [ 0.034304] *** VALIDATE cgroup *** [ 0.035015] *** VALIDATE cgroup2 *** [ 0.036314] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039036] Spectre V2 : User space: Vulnerable [ 0.040015] Speculative Store Bypass: Vulnerable [ 0.043015] debug: unmapping init [mem 0xffffffffbc859000-0xffffffffbc860fff] [ 0.046148] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049026] ... bit width: 48 [ 0.050009] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053010] ... fixed-purpose events: 3 [ 0.054009] ... event mask: 000000070000000f [ 0.055325] rcu: Hierarchical SRCU implementation. [ 0.057247] smp: Bringing up secondary CPUs ... [ 0.058502] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062172] smp: Brought up 1 node, 4 CPUs [ 0.064010] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141533] node 0 deferred pages initialised in 74ms [ 0.145012] devtmpfs: initialized [ 0.146309] x86/mm: Memory block size: 128MB [ 0.148867] gcov: version magic: 0x41383552 [ 0.150324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152245] pinctrl core: initialized pinctrl subsystem [ 0.153226] [ 0.153754] ************************************************************* [ 0.154017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155015] ** ** [ 0.156015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157016] ** ** [ 0.158017] ** This means that this kernel is built to expose internal ** [ 0.159017] ** IOMMU data structures, which may compromise security on ** [ 0.160022] ** your system. ** [ 0.161015] ** ** [ 0.162032] ** If you see this message and you are not debugging the ** [ 0.163015] ** kernel, report this immediately to your vendor! ** [ 0.164020] ** ** [ 0.165018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166024] ************************************************************* [ 0.167932] NET: Registered protocol family 16 [ 0.168480] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172013] cpuidle: using governor menu [ 0.173598] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175465] PCI: Using configuration type 1 for base access [ 0.178151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.187111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193052] cryptd: max_cpu_qlen set to 1000 [ 0.195232] ACPI: Added _OSI(Module Device) [ 0.197035] ACPI: Added _OSI(Processor Device) [ 0.198011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204296] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211426] ACPI: Interpreter enabled [ 0.213086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215017] ACPI: Using IOAPIC for interrupt routing [ 0.216165] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220484] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230401] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242359] acpiphp: Slot [2] registered [ 0.243128] acpiphp: Slot [5] registered [ 0.245167] acpiphp: Slot [6] registered [ 0.246135] acpiphp: Slot [7] registered [ 0.248140] acpiphp: Slot [8] registered [ 0.249174] acpiphp: Slot [9] registered [ 0.251123] acpiphp: Slot [10] registered [ 0.253140] acpiphp: Slot [3] registered [ 0.255124] acpiphp: Slot [4] registered [ 0.256122] acpiphp: Slot [11] registered [ 0.258120] acpiphp: Slot [12] registered [ 0.260154] acpiphp: Slot [13] registered [ 0.261141] acpiphp: Slot [14] registered [ 0.263124] acpiphp: Slot [15] registered [ 0.265164] acpiphp: Slot [16] registered [ 0.266160] acpiphp: Slot [17] registered [ 0.268122] acpiphp: Slot [18] registered [ 0.270124] acpiphp: Slot [19] registered [ 0.271152] acpiphp: Slot [20] registered [ 0.273117] acpiphp: Slot [21] registered [ 0.275149] acpiphp: Slot [22] registered [ 0.277142] acpiphp: Slot [23] registered [ 0.278104] acpiphp: Slot [24] registered [ 0.280127] acpiphp: Slot [25] registered [ 0.281122] acpiphp: Slot [26] registered [ 0.282099] acpiphp: Slot [27] registered [ 0.283114] acpiphp: Slot [28] registered [ 0.284148] acpiphp: Slot [29] registered [ 0.286092] acpiphp: Slot [30] registered [ 0.287098] acpiphp: Slot [31] registered [ 0.288058] PCI host bridge to bus 0000:00 [ 0.289017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306207] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.310450] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320755] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336589] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339571] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.345570] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.369020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374931] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.391020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436853] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.446023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489484] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.533023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549985] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.564020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593483] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.603017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.620024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.652030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670525] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.683015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.690019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.710016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.723627] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727489] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.730485] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.732425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735349] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740228] iommu: Default domain type: Passthrough [ 0.742675] SCSI subsystem initialized [ 0.744148] ACPI: bus type USB registered [ 0.746143] usbcore: registered new interface driver usbfs [ 0.748101] usbcore: registered new interface driver hub [ 0.750116] usbcore: registered new device driver usb [ 0.752266] pps_core: LinuxPPS API ver. 1 registered [ 0.754013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.758104] PTP clock support registered [ 0.760126] EDAC MC: Ver: 3.0.0 [ 0.762211] PCI: Using ACPI for IRQ routing [ 0.763603] NetLabel: Initializing [ 0.765013] NetLabel: domain hash size = 128 [ 0.766010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.768083] NetLabel: unlabeled traffic allowed by default [ 0.770146] vgaarb: loaded [ 0.771384] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.773021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.781726] clocksource: Switched to clocksource kvm-clock [ 0.900873] VFS: Disk quotas dquot_6.6.0 [ 0.903035] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.906263] *** VALIDATE ramfs *** [ 0.907946] *** VALIDATE hugetlbfs *** [ 0.910913] pnp: PnP ACPI init [ 0.913913] pnp: PnP ACPI: found 6 devices [ 0.933995] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.937184] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.939288] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.941463] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.944088] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.946581] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.949527] NET: Registered protocol family 2 [ 0.952264] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.957638] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.961425] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.969067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.973693] TCP: Hash tables configured (established 65536 bind 65536) [ 0.977635] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.981867] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985337] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.988497] NET: Registered protocol family 1 [ 0.992240] RPC: Registered named UNIX socket transport module. [ 0.994486] RPC: Registered udp transport module. [ 0.996264] RPC: Registered tcp transport module. [ 0.997992] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.000731] NET: Registered protocol family 44 [ 1.002774] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.005097] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.007337] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.009897] PCI: CLS 0 bytes, default 64 [ 1.011739] Unpacking initramfs... [ 2.476129] debug: unmapping init [mem 0xffff96563cc54000-0xffff96563ffbffff] [ 2.480462] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.482951] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.486212] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.050720] Initialise system trusted keyrings [ 3.052346] Key type blacklist registered [ 3.054300] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.065986] zbud: loaded [ 3.069453] *** VALIDATE nfs *** [ 3.070520] *** VALIDATE nfs4 *** [ 3.071660] pstore: using deflate compression [ 3.075589] Platform Keyring initialized [ 3.201222] NET: Registered protocol family 38 [ 3.204951] Key type asymmetric registered [ 3.206841] Asymmetric key parser 'x509' registered [ 3.208487] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.211567] io scheduler mq-deadline registered [ 3.213354] io scheduler kyber registered [ 3.215521] io scheduler bfq registered [ 3.217523] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.220712] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.223622] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.226372] ACPI: Power Button [PWRF] [ 3.233528] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.239854] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.253371] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.266304] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.295822] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.324765] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.355646] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.360512] Non-volatile memory driver v1.3 [ 3.361560] Linux agpgart interface v0.103 [ 3.392495] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.395876] vda: detected capacity change from 0 to 75100160 [ 3.426906] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.430469] vdb: detected capacity change from 0 to 1073741824 [ 3.457121] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.460313] vdc: detected capacity change from 0 to 2621440000 [ 3.485241] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.488759] vdd: detected capacity change from 0 to 2621440000 [ 3.504130] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.506653] vde: detected capacity change from 0 to 4294967296 [ 3.519698] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.521424] vdf: detected capacity change from 0 to 4294967296 [ 3.530701] libphy: Fixed MDIO Bus: probed [ 3.535293] usbcore: registered new interface driver usbserial_generic [ 3.536925] usbserial: USB Serial support registered for generic [ 3.538431] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.541644] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.542680] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.544190] mousedev: PS/2 mouse device common for all mice [ 3.546083] rtc_cmos 00:05: RTC can wake from S4 [ 3.547809] rtc_cmos 00:05: registered as rtc0 [ 3.548896] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.551701] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.556386] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.556713] intel_pstate: CPU model not supported [ 3.560180] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.563614] hid: raw HID events driver (C) Jiri Kosina [ 3.564912] usbcore: registered new interface driver usbhid [ 3.566146] usbhid: USB HID core driver [ 3.567085] drop_monitor: Initializing network drop monitor service [ 3.568444] Initializing XFRM netlink socket [ 3.569799] NET: Registered protocol family 10 [ 3.571644] Segment Routing with IPv6 [ 3.572558] NET: Registered protocol family 17 [ 3.573595] mpls_gso: MPLS GSO support [ 3.577449] RAS: Correctable Errors collector initialized. [ 3.578634] AVX version of gcm_enc/dec engaged. [ 3.579488] AES CTR mode by8 optimization enabled [ 3.665559] sched_clock: Marking stable (3665480269, 0)->(4639925755, -974445486) [ 3.668748] registered taskstats version 1 [ 3.670981] Loading compiled-in X.509 certificates [ 3.673021] zswap: loaded using pool lzo/zbud [ 3.696895] Key type big_key registered [ 3.709139] Key type encrypted registered [ 3.710916] ima: No TPM chip found, activating TPM-bypass! [ 3.713156] ima: Allocated hash algorithm: sha1 [ 3.715038] ima: No architecture policies found [ 3.716985] evm: Initialising EVM extended attributes: [ 3.718972] evm: security.selinux [ 3.720241] evm: security.ima [ 3.721327] evm: security.capability [ 3.722599] evm: HMAC attrs: 0x1 [ 3.726683] rtc_cmos 00:05: setting system clock to 2026-09-05 04:18:44 UTC (1788581924) [ 3.733195] debug: unmapping init [mem 0xffffffffbd803000-0xffffffffbd9fffff] [ 3.736348] debug: unmapping init [mem 0xffffffffbc582000-0xffffffffbc858fff] [ 3.746068] Write protecting the kernel read-only data: 28672k [ 3.749604] debug: unmapping init [mem 0xffffffffbac03000-0xffffffffbadfffff] [ 3.752789] debug: unmapping init [mem 0xffffffffbb514000-0xffffffffbb5fffff] [ 3.793470] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.802679] systemd[1]: Detected virtualization kvm. [ 3.804691] systemd[1]: Detected architecture x86-64. [ 3.806834] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.831228] systemd[1]: No hostname configured. [ 3.833102] systemd[1]: Set hostname to . [ 3.835357] random: systemd: uninitialized urandom read (16 bytes read) [ 3.837987] systemd[1]: Initializing machine ID from random generator. [ 3.894562] random: ln: uninitialized urandom read (6 bytes read) [ 3.970831] random: systemd: uninitialized urandom read (16 bytes read) [ 3.973875] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.980537] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.984313] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.575805] device-mapper: uevent: version 1.0.3 [ 4.578620] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.341728] random: fast init done [ 5.362939] virtio_net virtio0 ens2: renamed from eth0 [ 5.410412] scsi host0: ata_piix [ 5.490678] scsi host1: ata_piix [ 5.492291] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.495088] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.243973] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.229473] random: crng init done [ 10.230907] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.744946] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.948636] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.204450] SELinux: Disabled at runtime. [ 12.269260] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.278470] systemd[1]: Detected virtualization kvm. [ 12.280173] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.784397] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.787265] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.791610] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.797391] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.801255] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.809983] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.818168] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.989893] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.376224] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.713593] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.728163] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.876481] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.889345] EDAC sbridge: Ver: 1.1.2 [ 15.667068] Key type dns_resolver registered [ 16.002347] NFS: Registering the id_resolver key type [ 16.004557] Key type id_resolver registered [ 16.006127] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg251-server login: [ 44.904696] hrtimer: interrupt took 10682902 ns [ 50.147945] libcfs: loading out-of-tree module taints kernel. [ 50.182629] Key type ._llcrypt registered [ 50.185361] Key type .llcrypt registered [ 50.303161] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 70.892494] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 73.797176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.833615] alg: No test for adler32 (adler32-zlib) [ 75.768490] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 77.025493] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 78.984544] Key type lgssc registered [ 82.031918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.789156] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.550986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 162.758522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.792773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.085829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.117458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.237302] Lustre: lustre-MDT0000: new disk, initializing [ 164.329632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.344911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.264405] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.715415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.799506] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.846880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.854847] Lustre: Skipped 1 previous similar message [ 184.945732] Lustre: lustre-MDT0001: new disk, initializing [ 185.040547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.062308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.070541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.848939] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.067821] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.729802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.132611] Lustre: lustre-OST0000: new disk, initializing [ 207.144510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.154307] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.229472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.557741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.577185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.611268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.777514] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.265529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.399805] Lustre: lustre-OST0001: new disk, initializing [ 229.406740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.414295] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.495164] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.126758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.132376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.218584] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.297762] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.117532] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.938629] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.991863] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing check_logdir /tmp/testlogs/ [ 267.406353] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing yml_node [ 272.336337] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 275.025071] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 277.735915] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 279.247124] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 00:23:18 EDT 2026 [ 298.103530] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 300.098691] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.389190] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.401117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.402447] Lustre: Skipped 1 previous similar message [ 316.417192] Lustre: Skipped 3 previous similar messages [ 321.503830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.648664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.661441] Lustre: Skipped 6 previous similar messages [ 330.207248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 330.413148] Lustre: server umount lustre-MDT0000 complete [ 334.445770] LustreError: 9517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582255 with bad export cookie 10168165014116261850 [ 334.459075] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.481644] LustreError: 9517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 336.864569] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.865229] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.877925] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 336.890202] Lustre: Skipped 2 previous similar messages [ 336.893677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.897244] Lustre: Skipped 1 previous similar message [ 339.872532] LustreError: 9481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.881908] LustreError: 9481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 340.924633] Lustre: server umount lustre-MDT0001 complete [ 344.523607] Lustre: server umount lustre-OST0000 complete [ 348.090267] Lustre: server umount lustre-OST0001 complete [ 355.062374] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 363.421899] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 404.991672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.271622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.310356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.390722] Lustre: lustre-MDT0000: new disk, initializing [ 405.473092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.492734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.258818] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 421.003389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.101927] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.151629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.154656] Lustre: Skipped 1 previous similar message [ 421.259526] Lustre: lustre-MDT0001: new disk, initializing [ 421.359432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.392850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.403892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.920809] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 434.519503] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.900474] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 445.960420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 446.291474] Lustre: lustre-OST0000: new disk, initializing [ 446.302186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.308061] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 446.430094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.415402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 447.424851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 447.673522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.394618] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 461.520260] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.734611] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.033315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.953701] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.165675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 478.202352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.216535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.231577] Lustre: Skipped 1 previous similar message [ 478.237285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.243047] Lustre: Skipped 2 previous similar messages [ 478.703462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.587539] Lustre: server umount lustre-OST0000 complete [ 487.906511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.914940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.928185] Lustre: Skipped 1 previous similar message [ 487.934407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.937020] Lustre: Skipped 1 previous similar message [ 491.451134] Lustre: server umount lustre-MDT0000 complete [ 494.053609] LustreError: 15873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.078432] LustreError: 15873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 495.253532] LustreError: 15867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582416 with bad export cookie 10168165014116263383 [ 495.267894] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.567474] Lustre: server umount lustre-MDT0001 complete [ 500.449409] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 00:27:00 (1788582420) [ 506.863347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.443613] LustreError: 19304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.550312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.902678] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 512.999614] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.484048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.548489] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 533.854978] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 539.816728] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.225458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 554.988426] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 563.403727] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.237491] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 577.513416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.528134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.543450] Lustre: Skipped 1 previous similar message [ 577.548544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 577.554064] Lustre: Skipped 2 previous similar messages [ 582.440489] Lustre: server umount lustre-MDT0000 complete [ 585.699305] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.722767] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 586.574580] LustreError: 19285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582507 with bad export cookie 10168165014116264265 [ 586.584366] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.589907] LustreError: 19285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 587.052456] Lustre: server umount lustre-MDT0001 complete [ 600.500583] Lustre: server umount lustre-OST0000 complete [ 621.537328] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 632.351171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.875132] LustreError: 23685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.968926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 632.977584] Lustre: Skipped 2 previous similar messages [ 637.323614] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 645.731473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.629520] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 653.857921] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 660.654993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.769490] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 676.419791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 676.581241] Lustre: lustre-OST0001: new disk, initializing [ 676.587817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 676.594510] Lustre: 26001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 678.515395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 678.522337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 678.631359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 683.526335] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 693.587533] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 697.361403] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 712.673389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.688088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.707070] Lustre: Skipped 3 previous similar messages [ 712.729153] Lustre: Skipped 5 previous similar messages [ 716.307722] Lustre: server umount lustre-MDT0000 complete [ 717.799189] LustreError: 24415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.823953] LustreError: 24415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 720.310850] LustreError: 24824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582641 with bad export cookie 10168165014116265112 [ 720.319667] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 720.668478] Lustre: server umount lustre-MDT0001 complete [ 734.838464] Lustre: server umount lustre-OST0000 complete [ 738.273380] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788582643/real 1788582643] req@ffff9656b1629f80 x1875464156339200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788582659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 738.294472] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.306314] Lustre: Skipped 2 previous similar messages [ 738.511759] Lustre: server umount lustre-OST0001 complete [ 756.502511] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 767.952937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.552152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.558214] Lustre: Skipped 3 previous similar messages [ 773.992904] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 783.563832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.841117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 789.906834] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 793.094342] Lustre: 30177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 799.237373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 805.004890] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 808.754844] LustreError: 30529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.776958] LustreError: 30529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 816.291582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 825.878853] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 835.090624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 839.275348] Lustre: 32056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 845.792176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 845.796954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.189749] Lustre: server umount lustre-MDT0000 complete [ 855.350743] LustreError: 29008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582776 with bad export cookie 10168165014116266407 [ 855.361025] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.370496] LustreError: 29008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 855.648304] Lustre: server umount lustre-MDT0001 complete [ 870.689396] Lustre: server umount lustre-OST0000 complete [ 873.759737] Lustre: 3638:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788582778/real 1788582778] req@ffff9656b51ba300 x1875464156391552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788582794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 874.834628] Lustre: server umount lustre-OST0001 complete [ 883.496992] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 00:33:22 (1788582802) [ 910.445253] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 919.640348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.202913] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 934.820048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.933800] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 942.391659] Lustre: 36006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 950.189512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.575750] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 960.944631] LustreError: 36359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.975913] LustreError: 36359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 966.801491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 972.838106] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 979.637392] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 982.961434] Lustre: 37884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 991.723361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.731606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.745280] Lustre: Skipped 4 previous similar messages [ 991.750614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.755225] Lustre: Skipped 5 previous similar messages [ 994.174834] Lustre: server umount lustre-MDT0000 complete [ 997.697923] LustreError: 34839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582918 with bad export cookie 10168165014116267513 [ 997.701855] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.709467] LustreError: 34839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 998.226797] Lustre: server umount lustre-MDT0001 complete [ 1004.412615] Lustre: server umount lustre-OST0000 complete [ 1007.941731] Lustre: server umount lustre-OST0001 complete [ 1016.613232] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 00:35:35 (1788582935) [ 1032.464034] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1042.076125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.529683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.537484] Lustre: Skipped 7 previous similar messages [ 1046.903306] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1054.387960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.540548] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1061.312720] Lustre: 41173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1066.872751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1074.022335] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1084.043134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.093136] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1103.009241] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1106.517105] Lustre: 43054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1107.617946] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1107.621020] Lustre: Skipped 1 previous similar message [ 1111.009016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1116.832628] Lustre: server umount lustre-MDT0000 complete [ 1119.880229] LustreError: 42364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583040 with bad export cookie 10168165014116268612 [ 1119.891413] LustreError: 42364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1119.892598] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.226540] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.230510] Lustre: Skipped 9 previous similar messages [ 1126.614574] Lustre: server umount lustre-MDT0001 complete [ 1134.551190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.799798] LustreError: 3636:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9656b7c1bb80 x1875464156492032/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1149.866587] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1154.476750] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1158.499884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.275152] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1164.301463] Lustre: Skipped 1 previous similar message [ 1164.386912] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1172.305178] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1173.800820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1179.497598] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1180.899480] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1189.862318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1192.115640] Lustre: server umount lustre-MDT0000 complete [ 1195.943494] LustreError: 40006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583116 with bad export cookie 10168165014116269718 [ 1195.945926] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1195.951905] LustreError: 40006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1196.444669] Lustre: server umount lustre-MDT0001 complete [ 1210.623489] Lustre: server umount lustre-OST0000 complete [ 1225.029848] Lustre: server umount lustre-OST0001 complete [ 1235.448823] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 00:39:14 (1788583154) [ 1253.492977] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1263.968146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.566592] LustreError: 47663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.588183] LustreError: 47663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 1270.079956] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1279.710722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.972298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1279.979339] Lustre: Skipped 21 previous similar messages [ 1285.116440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1288.381965] Lustre: 48816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1295.854492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1304.173717] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1312.470572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.097764] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1327.761974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1333.600888] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1354.495228] Lustre: server umount lustre-MDT0000 complete [ 1358.778422] LustreError: 50703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583279 with bad export cookie 10168165014116270537 [ 1358.792625] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.793270] LustreError: 50703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1359.089493] Lustre: server umount lustre-MDT0001 complete [ 1365.431865] Lustre: server umount lustre-OST0000 complete [ 1369.133262] Lustre: server umount lustre-OST0001 complete [ 1379.778922] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 00:41:39 (1788583299) [ 1382.164509] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1383.843854] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 00:41:43 (1788583303) [ 1393.145653] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 1400.269323] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1423.693153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.138735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1424.332996] Lustre: lustre-MDT0000: new disk, initializing [ 1424.521633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1429.150472] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1439.022287] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.244747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.591492] Lustre: lustre-OST0000: new disk, initializing [ 1446.596462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1446.604099] Lustre: Skipped 1 previous similar message [ 1446.611958] Lustre: 54890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1448.712876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1448.723080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1448.763615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1453.621859] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1463.889589] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1466.234553] Lustre: DEBUG MARKER: create test files [ 1500.147035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.158438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.177630] Lustre: Skipped 11 previous similar messages [ 1506.290313] Lustre: server umount lustre-OST0000 complete [ 1511.321742] Lustre: server umount lustre-MDT0000 complete [ 1524.043282] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1527.030209] Key type lgssc unregistered [ 1527.360841] LNet: 56520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.365811] LNetError: 56520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.380092] LNet: Removed LNI 192.168.202.151@tcp [ 1528.522185] Key type .llcrypt unregistered [ 1528.525274] Key type ._llcrypt unregistered [ 1558.021737] Key type ._llcrypt registered [ 1558.031333] Key type .llcrypt registered [ 1558.161731] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1559.496939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.755613] alg: No test for adler32 (adler32-zlib) [ 1560.886637] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1561.142107] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1562.839222] Key type lgssc registered [ 1564.106880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1573.487232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1573.516245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.912381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.236718] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1586.807945] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1593.640847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1594.060284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.182434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1600.337730] Lustre: 59249:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1600.352315] Lustre: 59249:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1600.362174] Lustre: 59249:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1600.380964] Lustre: 59249:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1600.394536] Lustre: 59249:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1600.410074] Lustre: 59249:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1601.724842] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1609.907981] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1616.545161] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1621.888766] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1631.202464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.210184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1631.224055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1635.479577] Lustre: server umount lustre-OST0000 complete [ 1639.457436] Lustre: server umount lustre-MDT0000 complete [ 1651.451440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1653.992898] Key type lgssc unregistered [ 1654.476132] LNet: 60858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.484860] LNetError: 60858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.504562] LNet: Removed LNI 192.168.202.151@tcp [ 1656.560743] Key type .llcrypt unregistered [ 1656.577089] Key type ._llcrypt unregistered [ 1689.801829] Key type ._llcrypt registered [ 1689.804367] Key type .llcrypt registered [ 1689.938597] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1691.257487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.300436] alg: No test for adler32 (adler32-zlib) [ 1692.521744] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1692.832724] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1694.559909] Key type lgssc registered [ 1695.681770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.202554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.218070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.707511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.304743] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1717.617705] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.887401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.181391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.414242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1729.525920] Lustre: 63687:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1729.549314] Lustre: 63687:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1729.560422] Lustre: 63687:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1729.574834] Lustre: 63687:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1729.591548] Lustre: 63687:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1729.604566] Lustre: 63687:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1732.161773] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1740.532311] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1745.899408] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1755.108213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.118546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.232850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.977701] Lustre: server umount lustre-OST0000 complete [ 1765.318822] Lustre: server umount lustre-MDT0000 complete [ 1777.417037] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 1780.517995] Key type lgssc unregistered [ 1780.856286] LNet: 65243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.868547] LNetError: 65243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1780.893356] LNet: Removed LNI 192.168.202.151@tcp [ 1781.850316] Key type .llcrypt unregistered [ 1781.853412] Key type ._llcrypt unregistered [ 1802.240404] Key type ._llcrypt registered [ 1802.242916] Key type .llcrypt registered [ 1802.443048] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 1820.015719] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 1820.988109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.167897] alg: No test for adler32 (adler32-zlib) [ 1822.502924] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 1822.764045] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 1824.527218] Key type lgssc registered [ 1825.769901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1865.619606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1865.684584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.074048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1867.171515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1867.267597] Lustre: lustre-MDT0000: new disk, initializing [ 1867.381799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.433733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1872.204642] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1884.381074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.595548] Lustre: 68391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.667836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1884.680928] Lustre: Skipped 1 previous similar message [ 1884.774181] Lustre: lustre-MDT0001: new disk, initializing [ 1884.878699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.956288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1884.973870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1890.119736] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1900.232946] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1906.286640] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1913.397617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.719369] Lustre: lustre-OST0000: new disk, initializing [ 1913.724908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1913.734344] Lustre: 70362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1913.875289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.471111] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 1920.632216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1920.660471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1920.846425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1930.025861] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1934.644389] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1934.879666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1940.009590] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1940.256865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1946.592580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.599696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.608505] LustreError: Skipped 1 previous similar message [ 1946.609264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.652744] Lustre: Skipped 1 previous similar message [ 1948.717726] Lustre: server umount lustre-OST0000 complete [ 1956.840219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.842975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.852451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.862140] Lustre: Skipped 1 previous similar message [ 1956.873308] Lustre: Skipped 1 previous similar message [ 1958.840334] Lustre: server umount lustre-MDT0000 complete [ 1962.863367] LustreError: 69188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583883 with bad export cookie 2801756788992770051 [ 1962.875371] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1962.880634] LustreError: 69188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1963.378883] Lustre: server umount lustre-MDT0001 complete [ 1972.363967] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 00:51:31 (1788583891) [ 1985.078356] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2023.703604] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 2033.877540] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2071.962053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.188664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2072.237225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2072.307876] Lustre: lustre-MDT0000: new disk, initializing [ 2072.375774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.388566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2077.674234] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2088.790672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.942233] Lustre: 76382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2088.992422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2089.000184] Lustre: Skipped 1 previous similar message [ 2089.062311] Lustre: lustre-MDT0001: new disk, initializing [ 2089.184118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.228861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2089.266272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2093.692232] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2102.873691] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2109.144304] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.057295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2116.480471] Lustre: lustre-OST0000: new disk, initializing [ 2116.484791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2116.493250] Lustre: 78348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2116.570385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.095585] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2126.918734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2126.942949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2127.063476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2133.726034] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2138.424925] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2138.627469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2143.166486] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2143.395360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2147.809464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.811655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.822476] LustreError: Skipped 1 previous similar message [ 2147.833022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.838301] Lustre: Skipped 1 previous similar message [ 2147.841444] Lustre: Skipped 1 previous similar message [ 2151.493181] Lustre: server umount lustre-OST0000 complete [ 2156.001982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2156.016194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.042256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.054258] Lustre: Skipped 1 previous similar message [ 2163.170727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.177776] Lustre: Skipped 3 previous similar messages [ 2169.311146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2169.562804] Lustre: server umount lustre-MDT0000 complete [ 2172.384277] LustreError: 76395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.413194] LustreError: 76395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2173.368492] LustreError: 76376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584094 with bad export cookie 2801756788992770940 [ 2173.372231] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.391806] LustreError: 76376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2173.746578] Lustre: server umount lustre-MDT0001 complete [ 2181.960890] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 00:55:01 (1788584101) [ 2188.069970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.591752] LustreError: 80763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.674716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.534309] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2193.890309] LustreError: 80764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.011460] LustreError: 80763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.429637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.384424] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2214.949152] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2220.325902] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2226.307396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2226.619348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.637867] Lustre: Skipped 1 previous similar message [ 2232.508957] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2239.899519] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2243.849934] Lustre: 83435:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 95d978c0-1794-411b-be96-255c506b82e5 at adminstrative request [ 2244.073658] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.088992] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2253.282966] Lustre: MGS: Client 95d978c0-1794-411b-be96-255c506b82e5 (at 0@lo) reconnecting [ 2253.292816] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x26e1d6f48af406ee to 0x26e1d6f48af4098e [ 2253.315433] Lustre: MGC192.168.202.151@tcp: Connection restored to 0@lo (at 0@lo) [ 2281.796284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2282.016119] Lustre: lustre-OST0001: new disk, initializing [ 2282.024074] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2282.038958] Lustre: 83828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2289.880179] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2290.854014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2290.873768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2291.087828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2302.030896] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2306.532630] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.561581] Lustre: Skipped 1 previous similar message [ 2306.568614] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.579786] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2306.597453] Lustre: Skipped 3 previous similar messages [ 2320.351204] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2320.623974] Lustre: server umount lustre-OST0001 complete [ 2321.908218] LustreError: 82585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.923460] LustreError: 82585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2325.985966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.997828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.041812] Lustre: Skipped 1 previous similar message [ 2326.052286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.068693] Lustre: Skipped 4 previous similar messages [ 2332.157903] LustreError: 82584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.192392] LustreError: 82584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2339.295128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2339.617244] Lustre: server umount lustre-OST0000 complete [ 2346.467686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2346.483893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.505111] Lustre: Skipped 1 previous similar message [ 2349.973812] Lustre: server umount lustre-MDT0000 complete [ 2352.610041] LustreError: 80763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.637503] LustreError: 80763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2354.166401] LustreError: 80744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584274 with bad export cookie 2801756788992772494 [ 2354.177684] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.182792] LustreError: 80744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2354.472257] Lustre: server umount lustre-MDT0001 complete [ 2364.207344] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 00:58:03 (1788584283) [ 2366.116976] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2368.007562] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 00:58:07 (1788584287) [ 2369.745299] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2371.734576] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 00:58:11 (1788584291) [ 2373.499157] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2375.383421] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 00:58:14 (1788584294) [ 2377.096518] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2378.883756] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 00:58:18 (1788584298) [ 2380.976853] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2387.593282] loop: module loaded [ 2387.627856] loop0: detected capacity change from 0 to 4294967296 [ 2387.737888] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2389.522923] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2393.767083] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2398.503425] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2424.979354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2425.053844] LustreError: 87955:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2425.068486] LustreError: 87955:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2425.087273] LustreError: 87955:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2425.099235] LustreError: 87955:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2447.236356] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 2458.065764] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 2499.804757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.144294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2500.186948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2500.343524] Lustre: lustre-MDT0000: new disk, initializing [ 2500.399283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.406553] Lustre: Skipped 1 previous similar message [ 2500.423398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2505.398348] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2516.639529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.779228] Lustre: 90823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2517.189816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2521.406295] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2531.591876] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2537.737104] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2545.697427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2546.175511] Lustre: 92807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2552.919232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2553.064941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2553.944090] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2563.741387] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2568.879522] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2569.145038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2574.390482] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2574.570455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2578.914619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.928302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.938109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.942283] Lustre: Skipped 2 previous similar messages [ 2578.945516] Lustre: Skipped 9 previous similar messages [ 2582.575731] Lustre: server umount lustre-OST0000 complete [ 2592.648568] Lustre: server umount lustre-MDT0000 complete [ 2594.273531] LustreError: 90835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.297213] LustreError: 90835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2596.373838] LustreError: 90816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584517 with bad export cookie 2801756788992773110 [ 2596.377692] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.387109] LustreError: 90816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2596.609734] Lustre: server umount lustre-MDT0001 complete [ 2605.178973] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 01:02:04 (1788584524) [ 2612.185434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.152193] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2627.250977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.217651] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2640.393458] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2647.087240] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2655.310913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.860366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.864751] Lustre: Skipped 4 previous similar messages [ 2663.434696] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2671.766719] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.504613] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2676.773828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2682.031132] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2682.263995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2690.197839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.415712] Lustre: lustre-OST0001: new disk, initializing [ 2690.418917] Lustre: Skipped 2 previous similar messages [ 2690.422711] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2690.426278] Lustre: Skipped 3 previous similar messages [ 2690.431759] Lustre: 98593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2697.333253] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2700.450135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2700.465589] Lustre: Skipped 2 previous similar messages [ 2700.468855] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2700.669198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2708.051131] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2713.767910] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2713.948619] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2719.175790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2719.417079] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2740.628840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2762.101427] Lustre: Failing over lustre-MDT0000 [ 2762.216834] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.232798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.241204] Lustre: Skipped 4 previous similar messages [ 2762.258899] Lustre: Skipped 6 previous similar messages [ 2762.651940] LustreError: 100489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.679350] LustreError: 100489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2762.969606] Lustre: server umount lustre-MDT0000 complete [ 2783.713286] Lustre: 66615:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788584688/real 1788584688] req@0000000032c4c12b x1875465987184000/t0(0) o400->MGC192.168.202.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788584704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2783.758755] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2789.165344] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2789.168737] LDISKFS-fs (dm-0): recovery complete [ 2789.186738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.463210] LustreError: 101413:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2794.468650] LustreError: 101413:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000007566b7d5 x1875465987189248/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788584714 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2794.479675] LustreError: 101413:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2794.978048] LustreError: 66613:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000037f980ab x1875465987192960/t0(0) o250->MGC192.168.202.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2795.407709] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2798.458723] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2798.463136] Lustre: 96228:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2800.613379] Lustre: 95949:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2800.629906] Lustre: 95949:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2800.645680] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2800.742933] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2800.857546] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2801.089140] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2801.429744] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2801.446185] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2801.779640] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2801.807743] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2801.826449] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2801.848871] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2802.513916] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2802.529864] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2802.851490] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2802.861759] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2804.152513] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2804.175110] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2804.823456] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2804.832967] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2805.159409] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2805.166769] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2808.371439] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2808.393648] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2809.041997] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2809.060353] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2809.383150] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2809.403658] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2816.523802] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2816.543816] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2817.197960] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2817.226384] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2817.559135] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2817.578510] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2832.800113] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2832.818788] Lustre: 101446:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2833.456469] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2833.472147] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2833.791221] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2833.806517] LustreError: 101446:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2858.063167] Lustre: 101446:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2858.078134] Lustre: 101446:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2858.089178] Lustre: 101446:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a50ed71b-a2f1-43de-8e2e-fcbef91e5054@192.168.202.51@tcp [ 2858.104798] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2858.113290] Lustre: 101446:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000c300c9c1 x1875465969144576/t0(8589934775) o101->a50ed71b-a2f1-43de-8e2e-fcbef91e5054@192.168.202.51@tcp:359/0 lens 592/0 e 0 to 0 dl 1788584789 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2858.147525] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2858.164320] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2858.170201] Lustre: lustre-MDT0000: Denying connection for new client a50ed71b-a2f1-43de-8e2e-fcbef91e5054 (at 192.168.202.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2858.229261] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2858.233412] Lustre: Skipped 2 previous similar messages [ 2858.238204] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2858.268600] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2858.272937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2875.896716] Lustre: server umount lustre-OST0000 complete [ 2879.973470] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.982424] LustreError: Skipped 2 previous similar messages [ 2892.780693] LustreError: 99063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.781034] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2892.806359] LustreError: 99063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 52 previous similar messages [ 2892.827271] Lustre: Skipped 8 previous similar messages [ 2894.303470] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2894.522141] Lustre: server umount lustre-OST0001 complete [ 2899.428345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.439401] Lustre: Skipped 5 previous similar messages [ 2905.042846] Lustre: server umount lustre-MDT0000 complete [ 2908.791551] LustreError: 95203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788584829 with bad export cookie 2801756788992824329 [ 2908.802116] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.125357] Lustre: server umount lustre-MDT0001 complete [ 2921.974240] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 01:07:20 (1788584840) [ 2931.897946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.903852] Lustre: *** cfs_fail_loc=197, val=0*** [ 2947.551691] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2952.671617] LustreError: 103431:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [ 2957.887515] Lustre: *** cfs_fail_loc=197, val=0*** [ 2957.890705] Lustre: Skipped 65 previous similar messages [ 2978.463311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.474825] Lustre: Skipped 2 previous similar messages [ 2985.230752] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 2993.466332] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2996.009320] Lustre: server umount lustre-OST0000 complete [ 3005.483538] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 01:08:44 (1788584924) [ 3028.173985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.155458] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3044.946588] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3049.642555] LustreError: 105083:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3049.650613] LustreError: 105083:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3049.658050] LustreError: 105083:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3050.464811] Lustre: server umount lustre-MDT0000 complete [ 3089.331589] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3102.347362] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3141.843950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.106373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3142.152229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3142.268958] Lustre: lustre-MDT0000: new disk, initializing [ 3142.433934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3147.794246] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3159.300920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.496305] Lustre: 109262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3159.901739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3166.076760] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3177.743065] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.929543] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3191.243038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.495654] Lustre: 111228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3198.512273] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3200.908335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3210.870778] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.296986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3219.740270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3225.051375] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3225.441940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3231.714388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.728091] LustreError: Skipped 1 previous similar message [ 3231.742417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.766200] Lustre: Skipped 1 previous similar message [ 3231.798313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.811203] Lustre: Skipped 4 previous similar messages [ 3234.655683] Lustre: server umount lustre-OST0000 complete [ 3245.412190] Lustre: server umount lustre-MDT0000 complete [ 3247.083224] LustreError: 111948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.100693] LustreError: 111948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3249.805519] LustreError: 109253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585170 with bad export cookie 2801756788992834962 [ 3249.813560] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.818795] LustreError: 109253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3250.285330] Lustre: server umount lustre-MDT0001 complete [ 3259.085944] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 01:12:58 (1788585178) [ 3268.720475] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3271.956132] Key type lgssc unregistered [ 3272.476155] LNet: 113717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3272.485205] LNetError: 113717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3272.509987] LNet: Removed LNI 192.168.202.151@tcp [ 3273.544104] Key type .llcrypt unregistered [ 3273.550951] Key type ._llcrypt unregistered [ 3293.929273] Key type ._llcrypt registered [ 3293.933026] Key type .llcrypt registered [ 3294.047646] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3312.869666] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3314.829765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3314.863846] alg: No test for adler32 (adler32-zlib) [ 3316.094674] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 3316.525232] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3318.375236] Key type lgssc registered [ 3319.712681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3364.438533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3364.471930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.713516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3365.745141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3365.855040] Lustre: lustre-MDT0000: new disk, initializing [ 3365.929257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.962743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3371.196509] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3383.179816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.563893] Lustre: lustre-OST0000: new disk, initializing [ 3383.567529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3383.573102] Lustre: Skipped 1 previous similar message [ 3383.577963] Lustre: 117890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3383.675933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.660775] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3393.193251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3393.209364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3393.239542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3400.819250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3400.991781] Lustre: lustre-OST0001: new disk, initializing [ 3400.995315] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3401.000688] Lustre: 118753:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3401.080239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3402.203159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3402.215789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3402.301624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3409.307635] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3422.693294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.697042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.703850] Lustre: Skipped 1 previous similar message [ 3422.720818] Lustre: Skipped 1 previous similar message [ 3425.667780] Lustre: server umount lustre-MDT0000 complete [ 3432.957246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3438.112799] LustreError: 117076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585358 with bad export cookie 13703438604097747832 [ 3438.126069] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.209524] Lustre: server umount lustre-OST0000 complete [ 3441.981778] Lustre: server umount lustre-OST0001 complete [ 3450.418638] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 01:16:10 (1788585370) [ 3477.736235] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 01:16:36 (1788585396) [ 3487.004872] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3509.607134] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 01:17:09 (1788585429) [ 3545.991351] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 3556.506738] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3595.660906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.012950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3596.046957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3596.141889] Lustre: lustre-MDT0000: new disk, initializing [ 3596.264461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.289176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3601.888695] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3614.113313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.268043] Lustre: 126508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3614.350704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3614.354169] Lustre: Skipped 1 previous similar message [ 3614.472061] Lustre: lustre-MDT0001: new disk, initializing [ 3614.598599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.646654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3614.678257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3619.971475] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3631.154102] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3637.565655] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3644.910935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.305863] Lustre: lustre-OST0000: new disk, initializing [ 3645.309779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3645.317062] Lustre: 128475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3645.424346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.293712] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3653.728677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.744718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3653.856618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3662.708971] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3667.586249] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3667.840691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3673.500419] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3673.729097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3679.716146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3679.717805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.728986] LustreError: Skipped 1 previous similar message [ 3679.731770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3679.739287] Lustre: Skipped 1 previous similar message [ 3679.759820] Lustre: Skipped 1 previous similar message [ 3681.986726] Lustre: server umount lustre-OST0000 complete [ 3686.376701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3686.380062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.390377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.953289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.957535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.987932] Lustre: server umount lustre-MDT0000 complete [ 3695.977464] LustreError: 126499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585616 with bad export cookie 13703438604097749057 [ 3695.994055] LustreError: 126499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3696.002733] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3696.316978] Lustre: server umount lustre-MDT0001 complete [ 3705.396617] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 01:20:24 (1788585624) [ 3712.733395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.205608] LustreError: 130888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.225895] LustreError: 130888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3713.271315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.217737] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3718.632270] LustreError: 130889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.747560] LustreError: 130888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.200295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.993561] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3742.854550] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.330470] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3756.597355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.932430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.953375] Lustre: Skipped 1 previous similar message [ 3763.597514] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3771.865213] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3780.222956] Lustre: *** cfs_fail_loc=159, val=0*** [ 3796.420575] Lustre: lustre-MDT0000: Client 884ed4a7-8a5d-40dd-aff0-a807355746f8 (at 192.168.202.51@tcp) reconnecting [ 3798.217255] Lustre: *** cfs_fail_loc=159, val=0*** [ 3798.236313] Lustre: Skipped 9 previous similar messages [ 3814.825871] Lustre: lustre-MDT0000: Client 884ed4a7-8a5d-40dd-aff0-a807355746f8 (at 192.168.202.51@tcp) reconnecting [ 3820.512313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.516156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.524056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.530255] Lustre: Skipped 1 previous similar message [ 3822.054266] Lustre: server umount lustre-OST0000 complete [ 3829.729711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.734700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3829.757709] Lustre: Skipped 1 previous similar message [ 3829.772877] Lustre: Skipped 1 previous similar message [ 3832.890615] Lustre: server umount lustre-MDT0000 complete [ 3834.848618] LustreError: 133097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.868614] LustreError: 133097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3836.921158] LustreError: 131623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788585757 with bad export cookie 13703438604097749939 [ 3836.923548] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.934928] LustreError: 131623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3847.396817] Lustre: server umount lustre-MDT0001 complete [ 3858.457947] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 3860.491169] Key type lgssc unregistered [ 3860.661523] LNet: 134667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3860.667172] LNetError: 134667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3860.685268] LNet: Removed LNI 192.168.202.151@tcp [ 3861.414687] Key type .llcrypt unregistered [ 3861.417138] Key type ._llcrypt unregistered [ 3890.158796] Key type ._llcrypt registered [ 3890.159878] Key type .llcrypt registered [ 3890.221376] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 3890.694421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3890.705323] alg: No test for adler32 (adler32-zlib) [ 3891.740632] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 3892.007384] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 3893.687441] Key type lgssc registered [ 3895.859562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.776786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3905.800420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.466465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.246895] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3917.803266] LustreError: 136079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.833321] LustreError: 136079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3921.381794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.783955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3926.532266] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3934.544269] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3941.392290] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3950.157769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3950.366237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.441498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3955.644865] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 3963.347257] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3975.490122] Lustre: *** cfs_fail_loc=159, val=0*** [ 3991.969982] Lustre: lustre-MDT0000: Client e8668524-12d0-45a3-a943-c81ec64a5632 (at 192.168.202.51@tcp) reconnecting [ 3995.118953] Lustre: *** cfs_fail_loc=159, val=0*** [ 4011.476508] Lustre: lustre-MDT0001: Client e8668524-12d0-45a3-a943-c81ec64a5632 (at 192.168.202.51@tcp) reconnecting [ 4013.431924] Lustre: *** cfs_fail_loc=159, val=0*** [ 4013.439036] Lustre: Skipped 3 previous similar messages [ 4029.938890] Lustre: lustre-MDT0001: Client e8668524-12d0-45a3-a943-c81ec64a5632 (at 192.168.202.51@tcp) reconnecting [ 4035.714711] Lustre: 139121:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4038.447524] Lustre: *** cfs_fail_loc=159, val=0*** [ 4038.455542] Lustre: Skipped 4 previous similar messages [ 4054.957193] Lustre: lustre-MDT0000: Client c3fc8e31-f6b7-4c95-9fdb-2d8820a6a298 (at 192.168.202.51@tcp) reconnecting [ 4056.553444] Lustre: *** cfs_fail_loc=159, val=0*** [ 4056.562357] Lustre: Skipped 14 previous similar messages [ 4072.376626] Lustre: lustre-MDT0000: Client c3fc8e31-f6b7-4c95-9fdb-2d8820a6a298 (at 192.168.202.51@tcp) reconnecting [ 4074.237641] Lustre: 139414:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4094.948405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4094.951598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.960087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4094.978826] Lustre: Skipped 1 previous similar message [ 4094.991051] Lustre: Skipped 1 previous similar message [ 4099.808865] Lustre: server umount lustre-OST0000 complete [ 4105.186019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.191554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.210282] Lustre: Skipped 1 previous similar message [ 4105.224792] Lustre: Skipped 1 previous similar message [ 4110.304105] LustreError: 136074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.306151] Lustre: server umount lustre-MDT0000 complete [ 4110.322161] LustreError: 136074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4114.402144] LustreError: 139182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.648601] LustreError: 136059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586035 with bad export cookie 15489954311766364515 [ 4114.661088] LustreError: 136059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4114.666565] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.022632] Lustre: server umount lustre-MDT0001 complete [ 4127.482100] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4130.515763] Key type lgssc unregistered [ 4130.786153] LNet: 140492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4130.793506] LNetError: 140492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4130.819935] LNet: Removed LNI 192.168.202.151@tcp [ 4131.923748] Key type .llcrypt unregistered [ 4131.925474] Key type ._llcrypt unregistered [ 4162.586685] Key type ._llcrypt registered [ 4162.592545] Key type .llcrypt registered [ 4162.766630] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4164.165559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4164.438804] alg: No test for adler32 (adler32-zlib) [ 4165.759395] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 4166.038258] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4167.783257] Key type lgssc registered [ 4169.633711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4178.261416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4178.284095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.702898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.441036] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4190.179608] LustreError: 141904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.200676] LustreError: 141904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4193.314919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.660314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4197.985478] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4205.060166] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4210.311623] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4216.918203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4217.356345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.431800] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4224.161520] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4224.558510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4231.793960] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4238.955670] Lustre: 144481:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4243.014422] Lustre: 144530:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4244.768667] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4245.628490] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4249.952934] Lustre: 144677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4250.966196] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4251.722871] Lustre: 144785:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4259.296089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4259.304756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.320307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4260.324169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4260.325870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4260.346149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.896610] Lustre: server umount lustre-OST0000 complete [ 4270.563494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.572179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.580215] Lustre: Skipped 1 previous similar message [ 4270.582490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4270.587331] Lustre: Skipped 2 previous similar messages [ 4271.865918] Lustre: server umount lustre-MDT0000 complete [ 4275.687788] LustreError: 141904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.712177] LustreError: 141904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4275.811698] LustreError: 141886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586196 with bad export cookie 16167336536619747722 [ 4275.819764] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4275.821609] LustreError: 141886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4276.132748] Lustre: server umount lustre-MDT0001 complete [ 4287.341394] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4291.130036] Key type lgssc unregistered [ 4291.515462] LNet: 145836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4291.524333] LNetError: 145836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4291.552027] LNet: Removed LNI 192.168.202.151@tcp [ 4292.452218] Key type .llcrypt unregistered [ 4292.454446] Key type ._llcrypt unregistered [ 4318.452569] Key type ._llcrypt registered [ 4318.458923] Key type .llcrypt registered [ 4318.580629] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4319.769331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4319.849983] alg: No test for adler32 (adler32-zlib) [ 4320.962755] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 4321.309080] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4322.927728] Key type lgssc registered [ 4324.525973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4334.000543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4334.023546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.546761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.043040] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4345.826946] LustreError: 147248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4345.858697] LustreError: 147248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4350.829328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.946783] LustreError: 147247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.304275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4356.106569] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4364.679415] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4371.566903] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4378.291803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.708777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.836812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4386.886496] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4395.232261] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4396.218294] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4401.084220] Lustre: *** cfs_fail_loc=159, val=0*** [ 4417.471548] Lustre: lustre-MDT0000: Client 7e3fb187-60dd-477b-97a4-3eed3dc6b97a (at 192.168.202.51@tcp) reconnecting [ 4427.235623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4427.237363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.247762] LustreError: Skipped 1 previous similar message [ 4427.248754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.289282] Lustre: Skipped 1 previous similar message [ 4430.313697] Lustre: server umount lustre-OST0000 complete [ 4437.473126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.475704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4437.501249] Lustre: Skipped 1 previous similar message [ 4437.509435] Lustre: Skipped 2 previous similar messages [ 4441.031342] Lustre: server umount lustre-MDT0000 complete [ 4442.593372] LustreError: 147247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.606789] LustreError: 147247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4445.364473] LustreError: 147229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586366 with bad export cookie 12094844488369265396 [ 4445.376018] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.383808] LustreError: 147229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4445.676649] Lustre: server umount lustre-MDT0001 complete [ 4457.869966] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4461.239735] Key type lgssc unregistered [ 4461.597271] LNet: 150909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4461.608819] LNetError: 150909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4461.633087] LNet: Removed LNI 192.168.202.151@tcp [ 4462.683650] Key type .llcrypt unregistered [ 4462.688335] Key type ._llcrypt unregistered [ 4491.347735] Key type ._llcrypt registered [ 4491.352796] Key type .llcrypt registered [ 4491.460397] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4492.915196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4492.998898] alg: No test for adler32 (adler32-zlib) [ 4494.120444] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 4494.358675] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4496.024326] Key type lgssc registered [ 4497.479226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4507.752058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4507.791536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.451814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.770838] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4519.913831] LustreError: 152319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.942385] LustreError: 152319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4520.929575] LustreError: 152320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.278637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.852630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4530.251942] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4538.241503] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.569097] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4551.757838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4552.134569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.254892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4558.365483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4566.086778] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4569.536113] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4591.692379] Lustre: lustre-MDT0001: evicting 824c6329-d4ed-4c32-a4e0-935284d50c26 (at 192.168.202.51@tcp) by administrative request [ 4591.708777] Lustre: Skipped 1 previous similar message [ 4620.781264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.782455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4620.807741] Lustre: Skipped 1 previous similar message [ 4620.809032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.833332] Lustre: Skipped 1 previous similar message [ 4625.891191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4631.019881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4631.031496] Lustre: Skipped 2 previous similar messages [ 4635.103256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4635.499262] Lustre: server umount lustre-OST0000 complete [ 4641.248834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.257611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.276539] Lustre: Skipped 1 previous similar message [ 4641.292734] Lustre: Skipped 1 previous similar message [ 4646.368725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.378845] Lustre: Skipped 1 previous similar message [ 4647.506771] Lustre: server umount lustre-MDT0000 complete [ 4651.489519] LustreError: 152319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.646029] LustreError: 152302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586572 with bad export cookie 1876965220306021144 [ 4651.647464] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4651.662910] LustreError: 152302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4652.175662] Lustre: server umount lustre-MDT0001 complete [ 4665.841098] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4669.476914] Key type lgssc unregistered [ 4669.759996] LNet: 156391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4669.787709] LNetError: 156391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4670.823602] LNet: Removed LNI 192.168.202.151@tcp [ 4672.010817] Key type .llcrypt unregistered [ 4672.013896] Key type ._llcrypt unregistered [ 4751.439574] Key type ._llcrypt registered [ 4751.443119] Key type .llcrypt registered [ 4751.683194] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4753.023446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4753.222376] alg: No test for adler32 (adler32-zlib) [ 4754.540516] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 4754.861425] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4756.575186] Key type lgssc registered [ 4758.173326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4766.270216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4766.294646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.856872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.748262] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4777.441873] LustreError: 158875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.464645] LustreError: 158875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4782.563230] LustreError: 158876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.661382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.156235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4788.474368] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4795.956823] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4801.442599] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4808.377042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.963544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4813.057297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4815.620329] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 4823.466215] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4827.383976] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4837.858702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4837.864106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.889915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4837.892338] Lustre: Skipped 1 previous similar message [ 4837.920755] Lustre: Skipped 1 previous similar message [ 4842.539196] Lustre: server umount lustre-OST0000 complete [ 4848.104245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.111249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4848.124745] Lustre: Skipped 1 previous similar message [ 4848.136567] Lustre: Skipped 1 previous similar message [ 4853.219588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.238449] Lustre: Skipped 1 previous similar message [ 4857.313500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.325809] Lustre: Skipped 1 previous similar message [ 4860.384494] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.655450] Lustre: server umount lustre-MDT0000 complete [ 4862.433840] LustreError: 158876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.454675] LustreError: 158876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4864.351080] LustreError: 158855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586785 with bad export cookie 12387731751336741734 [ 4864.358953] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.366528] LustreError: 158855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4864.687606] Lustre: server umount lustre-MDT0001 complete [ 4876.086646] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 4879.152282] Key type lgssc unregistered [ 4879.576811] LNet: 162463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4879.587452] LNetError: 162463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4879.609699] LNet: Removed LNI 192.168.202.151@tcp [ 4880.715988] Key type .llcrypt unregistered [ 4880.720339] Key type ._llcrypt unregistered [ 4907.060634] Key type ._llcrypt registered [ 4907.067917] Key type .llcrypt registered [ 4907.274872] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4908.440703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4908.473024] alg: No test for adler32 (adler32-zlib) [ 4909.570846] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 4909.853431] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 4911.609539] Key type lgssc registered [ 4912.799660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4928.646756] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 01:40:48 (1788586848) [ 4947.018691] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 4956.986233] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 4992.980353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4993.007895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.269251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4994.294575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4994.394155] Lustre: lustre-MDT0000: new disk, initializing [ 4994.539231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.563478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4999.610843] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5011.013741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.181711] Lustre: 166725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5011.259452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5011.265094] Lustre: Skipped 1 previous similar message [ 5011.341876] Lustre: lustre-MDT0001: new disk, initializing [ 5011.439608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5011.496230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5011.516410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5016.754483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5027.069701] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5033.495803] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5040.880032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5041.153206] Lustre: lustre-OST0000: new disk, initializing [ 5041.156403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5041.162713] Lustre: 168692:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5041.254577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.880425] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5049.446243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5049.464051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5049.566060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5058.084205] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5070.313159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5070.313211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.322200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5070.328235] LustreError: Skipped 1 previous similar message [ 5070.349642] Lustre: Skipped 1 previous similar message [ 5074.464029] Lustre: server umount lustre-OST0000 complete [ 5080.546657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.564856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5080.573135] Lustre: Skipped 1 previous similar message [ 5084.678567] Lustre: server umount lustre-MDT0000 complete [ 5088.153044] LustreError: 166716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587008 with bad export cookie 15371452248945438176 [ 5088.176249] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5088.180985] LustreError: 166716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5088.495426] Lustre: server umount lustre-MDT0001 complete [ 5099.791349] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 5102.934739] Key type lgssc unregistered [ 5103.221511] LNet: 170598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5103.231868] LNetError: 170598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5104.308282] LNet: Removed LNI 192.168.202.151@tcp [ 5105.154288] Key type .llcrypt unregistered [ 5105.156580] Key type ._llcrypt unregistered [ 5131.606498] Key type ._llcrypt registered [ 5131.608501] Key type .llcrypt registered [ 5131.689597] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 5133.245769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5133.263682] alg: No test for adler32 (adler32-zlib) [ 5134.364795] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 5134.652691] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 5136.362040] Key type lgssc registered [ 5137.796389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5146.063864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5146.079270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.455678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.633734] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5157.867724] LustreError: 172009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.892526] LustreError: 172009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5162.993387] LustreError: 172010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.100586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.333793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5168.894678] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5177.768256] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5184.434037] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5191.960694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.483080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.530539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5198.518675] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 5207.100919] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5214.293270] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5214.328278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.341098] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5214.351694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.372652] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5215.639329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5215.662185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.683906] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5215.692188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.703445] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5217.004570] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5217.036812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.044942] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5217.050830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.059402] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5217.921369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.069482] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5219.080350] Lustre: Skipped 1 previous similar message [ 5219.103462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.116316] Lustre: Skipped 1 previous similar message [ 5219.121335] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5219.127551] Lustre: Skipped 1 previous similar message [ 5219.135726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.157876] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5219.168864] Lustre: Skipped 1 previous similar message [ 5220.291139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.418297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.594425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.423425] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5223.427400] Lustre: Skipped 3 previous similar messages [ 5223.438410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.444068] Lustre: Skipped 3 previous similar messages [ 5223.448918] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5223.451898] Lustre: Skipped 3 previous similar messages [ 5223.454784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.461217] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5223.467036] Lustre: Skipped 3 previous similar messages [ 5224.505345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.501228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.753064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.769465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.769069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.819972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.744336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.845212] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5231.851335] Lustre: Skipped 7 previous similar messages [ 5231.865575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.876169] Lustre: Skipped 7 previous similar messages [ 5231.881544] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5231.887804] Lustre: Skipped 7 previous similar messages [ 5231.892260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.914914] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5231.917616] Lustre: Skipped 7 previous similar messages [ 5233.119838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.120726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.173334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.175290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.150348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.059840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.064523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.846067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.917768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.791360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.990983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.016042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.965337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.957768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.047311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.988193] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5247.994329] Lustre: Skipped 15 previous similar messages [ 5248.004584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5248.015228] Lustre: Skipped 15 previous similar messages [ 5248.026059] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5248.037381] Lustre: Skipped 15 previous similar messages [ 5248.041598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.056230] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5248.070058] Lustre: Skipped 15 previous similar messages [ 5249.021618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.807153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.937171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.931307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.101458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.993909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.847725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.253842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.288597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.186366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.094938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.969599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.600114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.548500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.423976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.593971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.497762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.645635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.482926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.671607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.979329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.198714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.965107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.142256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.192738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.299950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.233537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.015893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.825092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.922428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.870522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.785227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.626824] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5280.632157] Lustre: Skipped 32 previous similar messages [ 5280.660239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.676562] Lustre: Skipped 32 previous similar messages [ 5280.684693] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5280.694254] Lustre: Skipped 32 previous similar messages [ 5280.700883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.709852] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5280.717314] Lustre: Skipped 32 previous similar messages [ 5281.492240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.213338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.150486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.160040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.405236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.230088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.319312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.268370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.450176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.323706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.543895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.358431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.548156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.692331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.667328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.678154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.480075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.378870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.306257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.311825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.284029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.400567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.525140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.461471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.442732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.416457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.089402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.003628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.792307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.556815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.451689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.442432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.325535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.161117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.232725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.995643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.830354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.821733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.966241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.111710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.132372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.333777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.604203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.950737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.714943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.743991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.876514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.270726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.959669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.933223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.946638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.215509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.147653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.483787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.751379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.456192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.344978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.108705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.982686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.569168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.342918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.975857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.648131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.566625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.498378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.218042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.088794] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5345.095471] Lustre: Skipped 66 previous similar messages [ 5345.113558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.119082] Lustre: Skipped 66 previous similar messages [ 5345.120984] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5345.126622] Lustre: Skipped 66 previous similar messages [ 5345.132264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.143218] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5345.150776] Lustre: Skipped 66 previous similar messages [ 5345.929144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.788800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.586609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.491527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.342286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.232230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.031355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.067269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.684551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.402801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.032829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.775153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.406757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.215173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.996110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.716785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.464171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.321268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.091196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.018838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.887253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.568437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.268088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.906023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.696839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.587844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.657618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.700665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.416894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.483746] LustreError: 172029:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5369.506748] LustreError: 172029:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5369.509973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.449247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.425725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.430130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.546761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.609596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.698224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.524526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.825226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.737946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.548943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.515559] LustreError: 172029:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5380.528708] LustreError: 172029:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5380.543664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.401535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.455206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.253547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.215258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.129490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.142353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.060742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.083361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.131419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.974145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.009685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.828173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.722901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.530398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.492880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.307581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.261860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.983319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.980147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.884376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.824912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.656899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.423018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.264031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.161494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.194063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.330591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.364957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.416187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.233378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.055385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.113675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.985609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.028680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.947763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.903690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.789569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.798074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.772411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.734979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.562586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.402721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.476362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.471583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.401518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.331892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.348680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.419589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.238186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.093118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.009680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.169051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.105193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.112074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.032035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.025729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.867890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.671973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.409964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.184261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.991452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.777223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.639573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.598244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.527424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.778340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.079941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.963616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.026699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.937710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.923420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.841872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.939794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.856696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.791180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.524623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.586213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.374678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.214420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.952320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.886230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.611215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.577665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.468531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.383495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.278392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.271677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.163850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.112928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.847733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.699371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.615166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.566925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.405548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.121082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.975442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.682284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.643065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.480357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.444103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.355552] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5473.364282] Lustre: Skipped 141 previous similar messages [ 5473.385386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.398525] Lustre: Skipped 141 previous similar messages [ 5473.405791] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5473.408646] Lustre: Skipped 141 previous similar messages [ 5473.415483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.437121] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5473.446735] Lustre: Skipped 141 previous similar messages [ 5474.473559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.122976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.972475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.870983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.730501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.667664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.597641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.477950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.417237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.330100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.417058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.396853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.594463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.664361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.380666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.355394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.546415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.809517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.175508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.367573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.619662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.327380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.065144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.356819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.901932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.283712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.800409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.081633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.418039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.723194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.038403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.921917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.878663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.809684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.747347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.645842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.398506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.970762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.616992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.451428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.548080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.483994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.770419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.622990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.647223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.656445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.718890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.520055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.254747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.219246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.578845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.747996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.782716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.649502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.500325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.764507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.758118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.757669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.907809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.111120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.196474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.116173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.041294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.033581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.251826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.395083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.359724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.370732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.625139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.638639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.738255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.723158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.636269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.546122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.455218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.356308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.296922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.421196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.420979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.444824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.415151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.474983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.638522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.681532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.719554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.872403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.821265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.726269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.683167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.679071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.855132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.939791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.716904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.865796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.636056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.550470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.319182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.385264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.310537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.561706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.577097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.781922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.918138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.968295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.167951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.524563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.678031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.774560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.711664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.760706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.742665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.924412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.947650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.864047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.925871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.034901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.933142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.896324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.819886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.843445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.853798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.893813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.819677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.764080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.864189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.734497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.689202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.543677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.412030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.232874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.250268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.356471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.334636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.432285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.242080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.269957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.206739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.428660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.352273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.365572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.240574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.131593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.146572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.277282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.098823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.118593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.983157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.956343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.805401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.604428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.410229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.344846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.347703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.267539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.281899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.117707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.124666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.079394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.867805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.860967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.912622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.770958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.721736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.719766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.911489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.913023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.957599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.038756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.056254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.059917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.070286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.181695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.161395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.110945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.058968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.882395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.970564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.028489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.080658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.087460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.050553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.114690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.179625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.280211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.230732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.092615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.935703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.794354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.706979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.620712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.456586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.392510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.369856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.506407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.457376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.404618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.373590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.265905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.205751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.100431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.900541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.877562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.904769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.937078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.879412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.964610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.840271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.614214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.629889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.758629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.749514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.705086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.667399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.688276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.670252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.661922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.648525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.688605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.607279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.587391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.618930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.495376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.485618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.540977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.602828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.693467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.690222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.599736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.561911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.531691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.726676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.703745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.775853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.585221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.528293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.470468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.593551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.743513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.017154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.271494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.389917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.414467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.550693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.886252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.213716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.495431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.594811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.935830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.788823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.713788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.881755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.058812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.292842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.296246] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5730.302741] Lustre: Skipped 253 previous similar messages [ 5730.316717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.329689] Lustre: Skipped 253 previous similar messages [ 5730.333890] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5730.339384] Lustre: Skipped 253 previous similar messages [ 5730.347372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.357920] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5730.373466] Lustre: Skipped 253 previous similar messages [ 5731.483928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.856193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.955753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.095709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.323794] LustreError: 172029:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5736.342915] LustreError: 172029:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5736.360061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.472758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.351592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.545719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.444114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.453555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.803480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.386074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.825327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.025856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.097574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.372159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.460368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.699348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.496836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.601845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.680654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.751434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.979218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.878828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.936996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.825871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.868852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.976173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.248250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.156779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.246429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.389167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.432669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.656359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.764520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.564648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.664216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.696433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.725785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.788671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.784991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.745379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.720073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.760252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.608245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.599081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.738682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.710267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.711894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.576242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.632509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.635314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.811680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.788069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.831375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.656497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.651226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.817047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.779949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.767523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.742887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.876812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.899448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.008853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.061451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.012986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.021422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.929947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.020309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.984678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.994733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.969610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.115136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.095403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.159703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.283873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.357552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.410965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.542498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.767703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.760722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.729656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.670209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.844581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.735740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.810827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.731976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.745853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.700911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.602762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.576344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.552752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.409377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.613717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.585358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.542299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.374814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.241492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.100218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.014320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.125515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.279462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.167935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.334882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.440043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.521160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.616920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.915205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.033466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.962307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.043417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.085263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.234675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.440762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.579835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.596868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.727627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.729407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.843919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.771523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.867651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.891770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.927589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.077956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.119721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.169891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.459594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.496904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.357722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.433075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.541975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.680395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.806756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.885197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.730982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.706264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.558611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.635246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.660836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.736302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.636902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.575733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.558565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.591729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.814331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.756728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.758433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.850617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.739899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.960028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.159032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.359493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.256765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.282905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.189278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.150660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.041845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.193656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.186624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.093098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.269233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.245766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.154346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.070815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.062407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.022427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.209785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.175903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.250154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.043550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.066134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.154988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.023796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.895510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.752309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.617643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.616932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.502213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.333924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.310365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.230769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.083414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.043622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.953883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.802772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.708682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.547384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.416970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.356511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.086921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.995386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.988368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.959707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.926638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.938259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.691842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.601932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.476938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.353912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.095155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.986970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.954403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.992679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.839157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.601402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.453910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.460565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.383460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.535202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.479345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.328576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.284223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.216647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.120301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.069555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.041573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.172930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.078177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.188899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.110673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.044427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.247518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.293607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.349562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.466298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.346303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.407324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.424295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.614631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.693372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.783888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.974744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.961366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.206671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.495407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.723323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.684965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.643108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.381564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.330070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.244441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.250846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.120267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.196065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.919761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.691529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.476718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.469984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.361777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.387230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.452323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.480208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.492930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.599301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.757858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.894233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.871398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.693589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.732592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.826693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.978785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.124316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.157068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.004898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.264437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.632122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.076755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.630041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.852764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.939491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.151826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.275045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.206192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.336856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.435436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.425861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.195942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.962142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.980476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.848920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.089330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.386228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.502520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.142938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.098973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.812799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.622817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.784622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.855968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.663495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.777855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.871555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.810836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.820498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.698127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.640618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.493303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.636687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.666506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.627765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.568122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.725334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.911625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.857427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.911504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.741752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.660799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.525353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.650325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.732995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.762865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.945818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.981719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.020736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.881643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.903350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.649214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.709357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.892052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.977882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.929668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.206494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.393917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.614717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.579624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.703574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.652702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.607134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.603949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.573933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.463824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.364061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.283781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.081688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.151908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.159699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.167671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.305389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.089534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.153352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.049024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.065186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.976853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.902206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.979704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.997629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.922418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.071870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.008575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.926307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.813825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.663691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.687233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.741096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.721068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.782712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.804729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.810298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.816243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.811866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.622449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.514723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.414203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.403752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.472374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.544466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.582864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.582817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.542708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.590564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.629630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.392693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.081162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.944351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.765501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.910598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.783284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.856885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.814121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.118991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.197409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.358634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.600055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.168838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.271276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.435298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.589881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.065109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.347726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.133251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.902822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.730909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.957543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.258656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.575877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.704397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.742146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.024450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.193443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.105380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.065640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.921523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.714832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.784112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.631750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.496989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.658763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.657824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.728664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.595403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.629197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.702098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.950396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.190239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.389629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.409426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.361781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.580474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.616065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.599550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.767700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.927165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.969491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.134431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.202546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.211240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.515871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.345480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.419828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.374484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.456054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.485270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.351606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.342126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.601113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.719499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.709499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.806736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.754358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.691129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.509523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.396580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.417672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.290742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.333185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.383240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.385077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.346530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.311508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.283783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.283943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.250188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.074113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.044960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.105925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.958674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.909636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.039304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.983951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.054292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.969497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.015505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.972536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.983854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.811717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.043438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.900884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.101384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.255524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.216221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.291304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.435797] LustreError: 173923:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6546.448686] LustreError: 172029:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6992.874378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.904508] Lustre: Skipped 471 previous similar messages [ 6992.909579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.930237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6992.937414] LustreError: Skipped 3 previous similar messages [ 6993.606101] Lustre: server umount lustre-OST0000 complete [ 6997.989393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6998.003816] Lustre: Skipped 2 previous similar messages [ 7003.106486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.115021] Lustre: Skipped 1 previous similar message [ 7008.233026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.319136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7012.321341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.342077] Lustre: Skipped 2 previous similar messages [ 7012.738331] Lustre: server umount lustre-MDT0000 complete [ 7016.827264] LustreError: 171990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588937 with bad export cookie 8569404201686705078 [ 7016.831523] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.839584] LustreError: 171990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7017.459135] Lustre: server umount lustre-MDT0001 complete [ 7030.080354] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7033.008916] Key type lgssc unregistered [ 7033.386693] LNet: 227245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7033.397451] LNetError: 227245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7033.420406] LNet: Removed LNI 192.168.202.151@tcp [ 7034.483625] Key type .llcrypt unregistered [ 7034.488079] Key type ._llcrypt unregistered [ 7061.453362] Key type ._llcrypt registered [ 7061.457653] Key type .llcrypt registered [ 7061.578752] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7062.762628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7062.815901] alg: No test for adler32 (adler32-zlib) [ 7064.056907] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 7064.279640] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7065.951186] Key type lgssc registered [ 7067.285770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7075.085729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7075.101601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.445317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.647440] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7086.564624] LustreError: 228674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.583022] LustreError: 228674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7089.273026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.759758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.496524] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7102.158906] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7108.167886] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7114.919441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7115.173454] LustreError: 230495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.373196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.434749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7122.795654] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7130.228556] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7134.962802] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7136.972933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.992729] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7136.996330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7137.019451] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7137.024222] LustreError: 228694:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7140.840931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7140.853324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7140.873401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7141.860338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7141.865546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.526537] Lustre: server umount lustre-OST0000 complete [ 7157.095133] LustreError: 228694:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7180.813930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7181.288801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.329867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7182.343304] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7188.385104] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7191.611566] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7191.635663] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7197.286939] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.049451] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7205.151879] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9290440bd800.ost_server_uuid 50 [ 7206.745622] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9290440bd800.ost_server_uuid in IDLE state after 0 sec [ 7212.019032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7212.025560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.028982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7212.028992] Lustre: Skipped 2 previous similar messages [ 7212.076478] Lustre: Skipped 1 previous similar message [ 7217.134340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7217.152939] Lustre: Skipped 1 previous similar message [ 7218.148917] Lustre: server umount lustre-OST0000 complete [ 7222.757279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7222.767709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7222.781452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.620852] Lustre: server umount lustre-MDT0000 complete [ 7232.480305] LustreError: 229674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.493979] LustreError: 229674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7232.527840] LustreError: 228656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589153 with bad export cookie 16566430933618359796 [ 7232.533789] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7232.542722] LustreError: 228656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7232.995396] Lustre: server umount lustre-MDT0001 complete [ 7246.690968] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 7249.034967] Key type lgssc unregistered [ 7249.297525] LNet: 233890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7249.302257] LNetError: 233890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7249.333160] LNet: Removed LNI 192.168.202.151@tcp [ 7250.251793] Key type .llcrypt unregistered [ 7250.253820] Key type ._llcrypt unregistered [ 7283.236534] Key type ._llcrypt registered [ 7283.239241] Key type .llcrypt registered [ 7283.377214] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7284.429467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7284.565065] alg: No test for adler32 (adler32-zlib) [ 7285.564850] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 7285.922673] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 7287.695290] Key type lgssc registered [ 7288.849448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7299.934692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7299.952385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.402535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.457810] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7311.843554] LustreError: 235616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.864546] LustreError: 235616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7316.324033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.728556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7322.080574] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7325.443237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7332.165572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.648643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.798172] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7347.017080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7349.800572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7349.963441] Lustre: lustre-OST0001: new disk, initializing [ 7349.967643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7349.975482] Lustre: 237924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7350.116941] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.541134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7351.553265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7351.658876] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7356.614716] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7369.272118] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7379.975766] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7393.761377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7393.772845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.788399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7397.868191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7397.869727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.900122] Lustre: Skipped 2 previous similar messages [ 7399.080377] LustreError: 239031:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9655a7c55700) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7399.229421] Lustre: server umount lustre-MDT0000 complete [ 7402.981704] LustreError: 235616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.016743] LustreError: 235616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7403.873136] LustreError: 235599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589324 with bad export cookie 15952930256850183373 [ 7403.878328] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7403.899206] LustreError: 235599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7404.481804] Lustre: server umount lustre-MDT0001 complete [ 7418.561462] Lustre: server umount lustre-OST0000 complete [ 7433.453832] Lustre: server umount lustre-OST0001 complete [ 7454.676425] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7465.518271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.204114] LustreError: 241034:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7466.298395] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7470.747845] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7471.585689] LustreError: 241035:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.697242] LustreError: 241034:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.067675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.613433] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7485.880495] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7489.265766] Lustre: 242178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7495.780966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7502.031237] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7505.274112] LustreError: 242531:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.307461] LustreError: 242531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7505.317707] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7510.392585] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7511.423025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7511.644763] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7511.649945] Lustre: Skipped 1 previous similar message [ 7519.245964] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7529.142346] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7534.949712] Lustre: 244049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7547.375407] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7547.377685] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.383305] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.406565] Lustre: Skipped 1 previous similar message [ 7547.424881] Lustre: Skipped 6 previous similar messages [ 7552.487728] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7552.503819] Lustre: Skipped 3 previous similar messages [ 7553.033881] Lustre: server umount mylustre-MDT0000 complete [ 7556.452744] LustreError: 241011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589477 with bad export cookie 15952930256850186236 [ 7556.456899] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7556.460047] LustreError: 241011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7556.671666] Lustre: server umount mylustre-MDT0001 complete [ 7571.860634] Lustre: server umount mylustre-OST0000 complete [ 7573.983147] Lustre: 234877:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589478/real 1788589478] req@ffff96558a6a5f80 x1875471715434624/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788589494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7574.059142] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7574.086292] Lustre: Skipped 2 previous similar messages [ 7576.507575] Lustre: server umount mylustre-OST0001 complete [ 7602.297754] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7615.205136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.751052] LustreError: 246357:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.786802] LustreError: 246357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7615.900502] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.437386] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7630.111674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.221652] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7639.527513] Lustre: 247500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7647.074496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7655.004966] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7655.722345] LustreError: 247854:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.745260] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:161) [ 7655.745936] LustreError: 247854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7660.862665] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7664.550743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7670.274463] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 7672.916175] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7681.431923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7685.373229] Lustre: 249371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7695.848926] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7695.856909] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.859743] Lustre: Skipped 2 previous similar messages [ 7695.873711] Lustre: Skipped 3 previous similar messages [ 7699.312747] Lustre: server umount tfs-MDT0000 complete [ 7703.191582] LustreError: 246332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589623 with bad export cookie 15952930256850188189 [ 7703.199277] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7703.202804] LustreError: 246332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7703.656939] Lustre: server umount tfs-MDT0001 complete [ 7718.323628] Lustre: server umount tfs-OST0000 complete [ 7721.248537] Lustre: 234878:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589626/real 1788589626] req@ffff96559f240e00 x1875471715487488/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788589642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7721.292589] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.317604] Lustre: Skipped 1 previous similar message [ 7723.255296] Lustre: server umount tfs-OST0001 complete [ 7748.649979] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7760.486807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.181995] LustreError: 251625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.203042] LustreError: 251625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7761.293177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.303809] Lustre: Skipped 3 previous similar messages [ 7766.168854] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7775.016794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.914518] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7784.645657] Lustre: 252766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7792.027589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7799.913293] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7800.657641] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:193) [ 7805.820052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7808.557120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7814.146184] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7816.729052] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 7823.948883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7828.422616] Lustre: 254639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7837.116618] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 02:29:16 (1788589756) [ 7855.075990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.102777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.383380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7868.665726] Lustre: server umount lustre-MDT0000 complete [ 7872.859299] LustreError: 251601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589793 with bad export cookie 15952930256850190107 [ 7872.876887] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7875.560885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7875.568494] Lustre: Skipped 12 previous similar messages [ 7887.327394] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7887.943776] Lustre: server umount lustre-MDT0001 complete [ 7906.274058] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7906.541620] Lustre: server umount lustre-OST0000 complete [ 7926.239423] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7926.536928] Lustre: server umount lustre-OST0001 complete [ 7933.811793] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 7945.008150] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 7984.640643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.934645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7984.958066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7985.012623] Lustre: lustre-MDT0000: new disk, initializing [ 7985.163696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7985.168556] Lustre: Skipped 3 previous similar messages [ 7985.188240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7989.652790] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8004.059981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.362122] Lustre: 257829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8004.445439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8004.449586] Lustre: Skipped 1 previous similar message [ 8004.593185] Lustre: lustre-MDT0001: new disk, initializing [ 8004.772353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8004.780511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8010.426130] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8022.331031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8022.623575] Lustre: lustre-OST0000: new disk, initializing [ 8022.630767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8022.638922] Lustre: 259412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8024.373685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8024.416517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8024.533257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8028.935209] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8039.995550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8040.243057] Lustre: lustre-OST0001: new disk, initializing [ 8040.249861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8040.259868] Lustre: 260284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8042.156182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8042.183924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8042.333180] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8047.026647] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8062.314120] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 02:33:00 (1788589980) [ 8078.570922] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 02:33:18 (1788589998) [ 8093.674122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.712255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8093.715043] Lustre: Skipped 6 previous similar messages [ 8093.733039] Lustre: Skipped 5 previous similar messages [ 8099.867367] Lustre: server umount lustre-OST0000 complete [ 8103.924305] LustreError: 259396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.977511] LustreError: 259396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 8106.980963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8112.370177] Lustre: server umount lustre-MDT0000 complete [ 8116.669849] LustreError: 257821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590037 with bad export cookie 15952930256850191878 [ 8116.677562] LustreError: 257821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8116.681363] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8123.061746] Lustre: server umount lustre-MDT0001 complete [ 8135.899579] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8209.423675] Lustre: server umount lustre-OST0001 complete [ 8217.211218] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8234.347560] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8269.707974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.067348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8270.091348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8270.140955] Lustre: lustre-MDT0000: new disk, initializing [ 8270.194847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.200347] Lustre: Skipped 3 previous similar messages [ 8270.214810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8274.437967] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8284.283797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.396921] Lustre: 265140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8284.463350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8284.468242] Lustre: Skipped 1 previous similar message [ 8284.592239] Lustre: lustre-MDT0001: new disk, initializing [ 8284.857560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8284.867772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8289.250675] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8298.708612] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8304.559946] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8311.485050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.722980] Lustre: lustre-OST0000: new disk, initializing [ 8311.727356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8311.733215] Lustre: 267106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8313.283934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8313.296639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8313.423971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8318.112578] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8326.933237] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8333.798801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.798833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8333.805029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8333.805038] Lustre: Skipped 8 previous similar messages [ 8333.812137] Lustre: Skipped 5 previous similar messages [ 8333.846107] LustreError: Skipped 1 previous similar message [ 8339.407216] Lustre: server umount lustre-OST0000 complete [ 8349.980802] Lustre: server umount lustre-MDT0000 complete [ 8353.806735] LustreError: 265131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590274 with bad export cookie 15952930256850193355 [ 8353.818597] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.834064] LustreError: 265131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8354.324689] Lustre: server umount lustre-MDT0001 complete [ 8367.016198] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8370.317474] Key type lgssc unregistered [ 8370.728698] LNet: 268867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8370.740083] LNetError: 268867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8370.764711] LNet: Removed LNI 192.168.202.151@tcp [ 8371.564174] Key type .llcrypt unregistered [ 8371.569682] Key type ._llcrypt unregistered [ 8400.949887] Key type ._llcrypt registered [ 8400.952340] Key type .llcrypt registered [ 8401.135592] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8402.926306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8402.961222] alg: No test for adler32 (adler32-zlib) [ 8404.131905] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 8404.394732] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8406.119184] Key type lgssc registered [ 8407.592746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8416.980392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8417.014423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.660256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.505516] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8429.035217] LustreError: 270327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8429.066988] LustreError: 270327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8434.146080] LustreError: 270326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8434.245221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.931991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8440.147182] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8448.533918] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8453.472023] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8459.844397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8460.153919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8464.262694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8465.944142] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8474.062085] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8474.908810] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8479.713451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8479.739576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.770362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8480.809766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8483.272937] Lustre: server umount lustre-OST0000 complete [ 8490.994950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.005898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8491.018174] Lustre: Skipped 1 previous similar message [ 8491.040411] Lustre: Skipped 2 previous similar messages [ 8493.679946] Lustre: server umount lustre-MDT0000 complete [ 8496.106757] LustreError: 270327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8496.117170] LustreError: 270327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8497.596765] LustreError: 272170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590418 with bad export cookie 13803545589702401212 [ 8497.617036] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8498.460430] Lustre: server umount lustre-MDT0001 complete [ 8510.660174] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8512.779409] Key type lgssc unregistered [ 8513.028561] LNet: 273911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8513.033256] LNetError: 273911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8513.043553] LNet: Removed LNI 192.168.202.151@tcp [ 8513.526290] Key type .llcrypt unregistered [ 8513.528409] Key type ._llcrypt unregistered [ 8533.748648] Key type ._llcrypt registered [ 8533.755375] Key type .llcrypt registered [ 8533.874634] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8535.053708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8535.069792] alg: No test for adler32 (adler32-zlib) [ 8536.211269] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 8536.456826] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8538.087871] Key type lgssc registered [ 8539.580742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8551.423248] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8551.446756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8567.008702] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8572.127311] LustreError: 275104:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.151@tcp: failed processing log, type 4: rc = -110 [ 8598.816296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.772860] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8613.812614] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8615.790681] Lustre: server umount lustre-OST0000 complete [ 8635.975936] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 8639.490878] Key type lgssc unregistered [ 8639.999894] LNet: 276734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8640.006831] LNetError: 276734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8641.064409] LNet: Removed LNI 192.168.202.151@tcp [ 8641.954404] Key type .llcrypt unregistered [ 8641.958391] Key type ._llcrypt unregistered [ 8683.487935] Key type ._llcrypt registered [ 8683.490528] Key type .llcrypt registered [ 8683.603831] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8684.789844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8684.880000] alg: No test for adler32 (adler32-zlib) [ 8686.084354] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 8686.346251] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 8687.999396] Key type lgssc registered [ 8689.229669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8702.010727] loop0: detected capacity change from 0 to 204800000 [ 8702.222333] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8702.244043] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8702.679801] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8703.875955] loop0: detected capacity change from 0 to 204800000 [ 8704.071828] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8704.086351] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8704.382909] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8705.609312] loop0: detected capacity change from 0 to 204800000 [ 8705.802724] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8705.815864] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8705.906142] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8707.224219] loop0: detected capacity change from 0 to 204800000 [ 8707.400045] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8707.413239] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8707.636344] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8708.788500] loop0: detected capacity change from 0 to 204800000 [ 8708.827313] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8713.592413] loop0: detected capacity change from 0 to 204800000 [ 8713.620495] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8718.856808] loop0: detected capacity change from 0 to 204800000 [ 8718.895812] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8723.720457] loop0: detected capacity change from 0 to 204800000 [ 8723.786787] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8728.240952] loop0: detected capacity change from 0 to 204800000 [ 8728.402245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8728.460756] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.556586] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8728.565469] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8728.574563] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8728.582913] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8728.591651] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8728.600505] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8728.609218] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8728.617321] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8728.626605] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8728.643324] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8728.656637] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8728.676296] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8728.689973] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8728.718487] Lustre: 280148:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8735.487633] Lustre: server umount MGS complete [ 8736.537141] loop0: detected capacity change from 0 to 204800000 [ 8736.616440] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.214680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.265279] Lustre: lustre-MDT0000: Aborting client recovery [ 8738.210077] loop1: detected capacity change from 0 to 204800000 [ 8738.385913] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.537964] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8738.549853] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8738.572479] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8738.595558] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8738.611700] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8738.639296] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8738.665967] Lustre: 280582:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8738.708662] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8738.930663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8739.024293] Lustre: lustre-MDT0001: Aborting client recovery [ 8740.005419] loop2: detected capacity change from 0 to 204800000 [ 8740.163339] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8740.240529] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8740.259400] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8740.270245] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8740.291534] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8740.308948] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8740.324883] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8740.337654] Lustre: 280661:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8740.424436] Lustre: Found index 0 for lustre-OST0000, updating log [ 8740.466138] Lustre: 280680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8740.536783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8740.554772] Lustre: lustre-OST0000: Aborting recovery [ 8741.624901] loop3: detected capacity change from 0 to 204800000 [ 8741.782399] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8741.915716] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8741.931616] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8741.947664] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8741.961692] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8741.976687] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8741.989576] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8742.006300] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8742.039697] Lustre: Found index 1 for lustre-OST0001, updating log [ 8742.096869] Lustre: 280752:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8742.200028] Lustre: lustre-OST0001: Aborting recovery [ 8746.278579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8746.299889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8746.334282] LustreError: 280526:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8746.373698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8746.455973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8748.324987] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8748.327684] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8751.963580] LustreError: 280493:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 229us: rc = -2 [ 8752.542900] LustreError: 280494:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 624us: rc = -2 [ 8752.559737] LustreError: 280494:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8768.991995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8768.996126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.001421] Lustre: Skipped 2 previous similar messages [ 8769.019978] Lustre: Skipped 3 previous similar messages [ 8771.713772] Lustre: server umount lustre-MDT0000 complete [ 8772.882441] LustreError: 280477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590693 with bad export cookie 1482744487919917754 [ 8772.889909] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8772.905846] LustreError: 280477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8774.111971] LustreError: 280491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.124872] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.127532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8774.131651] LustreError: 280491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8774.167405] Lustre: Skipped 2 previous similar messages [ 8777.122326] LustreError: 280490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.125381] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8777.154355] LustreError: 280490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8777.179858] Lustre: Skipped 2 previous similar messages [ 8779.408908] Lustre: server umount lustre-MDT0001 complete [ 8786.646453] Lustre: server umount lustre-OST0000 complete [ 8802.271748] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8802.658700] Lustre: server umount lustre-OST0001 complete [ 8811.660389] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 02:45:30 (1788590730) [ 8844.449915] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 8856.416893] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 8896.851352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.238357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8897.288280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8897.436710] Lustre: lustre-MDT0000: new disk, initializing [ 8897.609301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.617978] Lustre: Skipped 1 previous similar message [ 8897.634720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8903.136418] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8915.316085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.509042] Lustre: 284822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8915.587540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8915.600874] Lustre: Skipped 1 previous similar message [ 8915.700255] Lustre: lustre-MDT0001: new disk, initializing [ 8915.880136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8915.938736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8915.951979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8921.262745] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8931.731279] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8937.749837] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8944.357338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.681554] Lustre: lustre-OST0000: new disk, initializing [ 8944.686504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8944.693680] Lustre: 286793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8944.792857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.407410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8951.416951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8951.581357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8952.382611] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 8962.261830] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8967.558960] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8980.617459] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8986.823324] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8991.200828] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9033.700469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.710160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9033.715772] Lustre: Skipped 1 previous similar message [ 9036.577379] Lustre: server umount lustre-OST0000 complete [ 9043.936037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9043.941228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.958496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9043.958506] Lustre: Skipped 1 previous similar message [ 9043.982622] Lustre: Skipped 1 previous similar message [ 9046.920962] Lustre: server umount lustre-MDT0000 complete [ 9049.058968] LustreError: 288302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.097164] LustreError: 288302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9050.876090] LustreError: 284816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590971 with bad export cookie 1482744487919920463 [ 9050.880031] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9050.894540] LustreError: 284816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9051.428672] Lustre: server umount lustre-MDT0001 complete [ 9060.066702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.799986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9069.557191] Lustre: server umount MGS complete [ 9081.811292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.383267] LustreError: 290708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.412542] LustreError: 290721:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9082.484573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9087.973240] LustreError: 290709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.314584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9097.982098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.216677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9098.227640] Lustre: Skipped 1 previous similar message [ 9103.218808] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9110.330272] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9117.262447] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9123.958589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.417777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.427934] Lustre: Skipped 1 previous similar message [ 9131.568055] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9139.019778] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9168.354904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9168.370552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9168.394323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9169.377786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9181.151578] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9181.606485] Lustre: server umount lustre-OST0000 complete [ 9189.859385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.890199] Lustre: Skipped 2 previous similar messages [ 9192.489539] Lustre: server umount lustre-MDT0000 complete [ 9194.976051] LustreError: 293350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.994213] LustreError: 293350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9197.197629] LustreError: 290688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591117 with bad export cookie 1482744487919922297 [ 9197.206561] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9197.217156] LustreError: 290688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9197.699142] Lustre: server umount lustre-MDT0001 complete [ 9209.727989] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9212.616675] Key type lgssc unregistered [ 9212.904825] LNet: 294654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9212.914528] LNetError: 294654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9213.989205] LNet: Removed LNI 192.168.202.151@tcp [ 9214.861378] Key type .llcrypt unregistered [ 9214.863613] Key type ._llcrypt unregistered [ 9250.645691] Key type ._llcrypt registered [ 9250.648951] Key type .llcrypt registered [ 9250.769985] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9267.485602] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9268.881717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9268.960549] alg: No test for adler32 (adler32-zlib) [ 9270.166672] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 9270.454885] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9272.183242] Key type lgssc registered [ 9273.443347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9312.006554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9312.031354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.221582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9313.243100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9313.364322] Lustre: lustre-MDT0000: new disk, initializing [ 9313.450065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.466678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9318.667215] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9330.016934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9330.212216] Lustre: 298611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9330.310125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9330.318209] Lustre: Skipped 1 previous similar message [ 9330.424264] Lustre: lustre-MDT0001: new disk, initializing [ 9330.582974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9330.682451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9330.718238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9335.116713] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9344.623925] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9350.802484] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9357.187269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9357.411863] Lustre: lustre-OST0000: new disk, initializing [ 9357.414521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9357.418281] Lustre: 300577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9357.484157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9364.061892] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9364.603977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9364.625102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9364.740865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9375.586474] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9384.211614] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9398.015209] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9400.037900] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9452.002731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9452.014361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9452.014742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9452.024374] LustreError: Skipped 1 previous similar message [ 9452.029786] Lustre: Skipped 1 previous similar message [ 9455.709135] Lustre: server umount lustre-OST0000 complete [ 9462.243059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.248484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9462.253722] Lustre: Skipped 2 previous similar messages [ 9462.268736] Lustre: Skipped 1 previous similar message [ 9466.495568] Lustre: server umount lustre-MDT0000 complete [ 9470.600624] LustreError: 298602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591391 with bad export cookie 13218223508656078160 [ 9470.608414] LustreError: 298602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9470.614278] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9471.086467] Lustre: server umount lustre-MDT0001 complete [ 9479.720418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9485.584758] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9491.017534] Lustre: server umount MGS complete [ 9503.883456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.292156] LustreError: 304398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.310894] LustreError: 304398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9504.343224] LustreError: 304411:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9504.441907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.766263] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9509.860125] LustreError: 304399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.988363] LustreError: 304398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9518.184365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.491613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9524.442752] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9533.836055] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9541.438291] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9549.640427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9549.992229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.106483] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9565.018214] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9600.499075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9600.519899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9600.560481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9605.603963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9605.609808] Lustre: Skipped 1 previous similar message [ 9606.145414] Lustre: server umount lustre-OST0000 complete [ 9610.720457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9610.722183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9610.743469] Lustre: Skipped 2 previous similar messages [ 9610.761434] Lustre: Skipped 2 previous similar messages [ 9616.841545] Lustre: server umount lustre-MDT0000 complete [ 9620.520021] LustreError: 304380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591541 with bad export cookie 13218223508656080001 [ 9620.542436] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9620.858922] Lustre: server umount lustre-MDT0001 complete [ 9632.738986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [ 9635.640068] Key type lgssc unregistered [ 9635.920607] LNet: 308364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9635.936752] LNetError: 308364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9635.953242] LNet: Removed LNI 192.168.202.151@tcp [ 9636.730892] Key type .llcrypt unregistered [ 9636.732793] Key type ._llcrypt unregistered [ 9660.201130] Key type ._llcrypt registered [ 9660.205939] Key type .llcrypt registered [ 9660.362799] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [ 9676.600795] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9678.292623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9678.348650] alg: No test for adler32 (adler32-zlib) [ 9679.553527] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [ 9679.916243] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [ 9681.655214] Key type lgssc registered [ 9683.028902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9708.031446] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [ 9724.353492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9724.394441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.783326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9725.830676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9725.910408] Lustre: lustre-MDT0000: new disk, initializing [ 9726.007298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9726.050476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9731.911986] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [ 9743.963662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9744.366189] Lustre: lustre-OST0000: new disk, initializing [ 9744.369757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9744.378734] Lustre: Skipped 1 previous similar message [ 9744.387628] Lustre: 312424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9744.476383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.675132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9750.683382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9750.729932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9752.383584] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10567.927293] Lustre: server umount lustre-MDT0000 complete [10571.395186] LustreError: 311600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592492 with bad export cookie 17236549891875878043 [10571.412321] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10582.200928] Lustre: server umount lustre-OST0000 complete [10605.244555] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [10614.111722] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10651.166575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.541262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10651.580093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10651.689833] Lustre: lustre-MDT0000: new disk, initializing [10651.816597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10651.834214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10657.517591] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10668.097256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.213384] Lustre: 316758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10668.251650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10668.254746] Lustre: Skipped 1 previous similar message [10668.304840] Lustre: lustre-MDT0001: new disk, initializing [10668.392666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10668.438489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10668.448433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10673.268496] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10683.082852] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10688.709871] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10695.368574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10695.748863] Lustre: lustre-OST0000: new disk, initializing [10695.753868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10695.766991] Lustre: 318724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10695.916872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10702.965325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10702.983805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10703.177264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10703.449620] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10712.152303] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10717.470707] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10717.725776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10722.993473] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10723.466855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10728.955077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10728.958393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.969609] LustreError: Skipped 1 previous similar message [10728.978062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10728.986932] Lustre: Skipped 1 previous similar message [10732.125208] Lustre: server umount lustre-OST0000 complete [10739.168125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.191182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10739.191301] Lustre: Skipped 1 previous similar message [10739.203995] Lustre: Skipped 2 previous similar messages [10742.904781] Lustre: server umount lustre-MDT0000 complete [10746.406786] LustreError: 316750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592667 with bad export cookie 17236549891878818862 [10746.425439] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10746.428306] LustreError: 316750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10746.856908] Lustre: server umount lustre-MDT0001 complete [10750.204895] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10752.068990] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 03:17:51 (1788592671) [10759.026468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10759.625466] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10759.638366] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10759.693266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10764.629768] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10764.791609] LustreError: 321037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.911688] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10773.520168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10774.259714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10779.438680] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10787.831151] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10793.665214] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10799.974654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10800.348508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10807.116144] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10814.624869] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10822.406696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10822.588716] Lustre: lustre-OST0001: new disk, initializing [10822.591634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10822.595867] Lustre: 323873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10822.674749] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10827.576764] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10828.944982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10828.951511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10829.013868] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10829.041270] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10830.049550] LustreError: 324307:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10831.077930] LustreError: 324307:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10831.091906] LustreError: 324307:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10833.130180] LustreError: 324307:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10833.138237] LustreError: 324307:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10837.282829] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10837.291528] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [10837.480975] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10837.642479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10842.948476] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff929060e64000.ost_server_uuid 50 [10844.686272] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff929060e64000.ost_server_uuid in FULL state after 0 sec [10845.493138] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10845.510529] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10849.888920] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff929060e64000.ost_server_uuid 50 [10851.668744] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff929060e64000.ost_server_uuid in FULL state after 0 sec [10853.688420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10868.626222] Lustre: lustre-OST0001: Not available for connect from 192.168.202.51@tcp (stopping) [10869.218577] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10869.224758] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10869.231114] LustreError: Skipped 1 previous similar message [10869.268682] Lustre: Skipped 1 previous similar message [10872.972964] Lustre: server umount lustre-OST0001 complete [10873.753304] LustreError: 322856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.488045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10880.488558] LustreError: 324302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10880.489177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10880.489182] LustreError: Skipped 1 previous similar message [10880.495657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10880.495665] Lustre: Skipped 4 previous similar messages [10880.499689] Lustre: Skipped 1 previous similar message [10880.534170] LustreError: 324302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10885.140408] Lustre: server umount lustre-OST0000 complete [10890.731742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.742023] Lustre: Skipped 1 previous similar message [10890.742204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.754961] Lustre: Skipped 2 previous similar messages [10895.390615] Lustre: server umount lustre-MDT0000 complete [10895.841073] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10895.863431] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10899.074989] LustreError: 323871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592819 with bad export cookie 17236549891878819744 [10899.077171] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.088299] LustreError: 323871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10899.636304] Lustre: server umount lustre-MDT0001 complete [10910.568249] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [10913.347371] Key type lgssc unregistered [10913.650245] LNet: 326562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10913.665049] LNetError: 326562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10914.735144] LNet: Removed LNI 192.168.202.151@tcp [10915.430711] Key type .llcrypt unregistered [10915.432674] Key type ._llcrypt unregistered [10938.736594] Key type ._llcrypt registered [10938.739563] Key type .llcrypt registered [10938.875683] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [10939.828476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10939.922902] alg: No test for adler32 (adler32-zlib) [10940.972358] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [10941.217429] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [10942.871219] Key type lgssc registered [10944.187368] Lustre: Echo OBD driver; http://www.lustre.org/ [10951.713432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10951.729529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.146347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10957.770375] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10962.411612] LustreError: 327983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.431875] LustreError: 327983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10965.914918] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10967.530321] LustreError: 327982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.640770] LustreError: 327983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10973.907626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10974.294649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10979.716770] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10989.552441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10989.843035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10996.545447] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [10998.085666] LustreError: 329662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.109150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11004.102717] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11004.397648] LustreError: 329664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.417465] LustreError: 329664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11013.434760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11013.736638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11018.768911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11021.130693] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11031.124271] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11039.227090] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11039.500417] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11141.622677] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11141.627078] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.639076] LustreError: Skipped 1 previous similar message [11141.642935] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11141.674769] Lustre: Skipped 1 previous similar message [11146.291904] Lustre: server umount lustre-OST0001 complete [11146.749568] LustreError: 329664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11146.767121] LustreError: 329664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11151.842656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11151.844766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.874174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11151.892177] Lustre: Skipped 1 previous similar message [11151.908969] Lustre: Skipped 1 previous similar message [11156.963083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11156.972594] Lustre: Skipped 1 previous similar message [11157.566843] Lustre: server umount lustre-OST0000 complete [11162.080510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.093602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11162.099143] Lustre: Skipped 1 previous similar message [11167.205807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11167.211355] Lustre: Skipped 1 previous similar message [11167.874070] Lustre: server umount lustre-MDT0000 complete [11171.582982] LustreError: 331648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593092 with bad export cookie 4369406452848270466 [11171.588152] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11171.596551] LustreError: 331648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11171.919799] Lustre: server umount lustre-MDT0001 complete [11184.203920] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11186.975545] Key type lgssc unregistered [11187.290822] LNet: 332923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11187.295168] LNetError: 332923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11187.313149] LNet: Removed LNI 192.168.202.151@tcp [11188.344116] Key type .llcrypt unregistered [11188.347880] Key type ._llcrypt unregistered [11234.898777] Key type ._llcrypt registered [11234.900637] Key type .llcrypt registered [11235.002833] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11236.160767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11236.195966] alg: No test for adler32 (adler32-zlib) [11237.452919] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [11237.844891] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [11239.535873] Key type lgssc registered [11240.689825] Lustre: Echo OBD driver; http://www.lustre.org/ [11250.602182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11250.615346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.235589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11256.919648] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11261.416231] LustreError: 335209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11261.445413] LustreError: 335209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11266.530675] LustreError: 335210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11266.591767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.911497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11272.639404] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11276.263512] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11283.811989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11284.216524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11290.871355] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11291.423142] LustreError: 336702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.558704] LustreError: 337088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.588035] LustreError: 337088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11296.597896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11299.934237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11300.156328] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11305.479360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11305.753719] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11315.875402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11322.857463] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11333.600336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11333.602977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.610513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.161730] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11336.168797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.171284] Lustre: Skipped 2 previous similar messages [11336.179993] Lustre: Skipped 3 previous similar messages [11338.734038] Lustre: server umount lustre-MDT0000 complete [11341.282302] LustreError: 336165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11341.301926] LustreError: 336165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11342.122806] LustreError: 336350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593262 with bad export cookie 2641977891307294705 [11342.124885] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11342.135805] LustreError: 336350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11342.400172] Lustre: server umount lustre-MDT0001 complete [11356.859906] Lustre: server umount lustre-OST0000 complete [11371.284704] Lustre: server umount lustre-OST0001 complete [11390.057757] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11403.517777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11403.954732] LustreError: 340481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11404.042481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11408.999935] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11418.222952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.664627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11423.959884] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11426.815431] Lustre: 341620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11433.712866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.958888] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11443.505441] LustreError: 341975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.537983] LustreError: 341975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11448.640041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11450.593994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11450.928611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11450.935994] Lustre: Skipped 1 previous similar message [11455.990335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11457.949453] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11466.134536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11470.498831] Lustre: 343494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11474.926718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11474.937237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11474.947611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11481.360929] Lustre: server umount lustre-MDT0000 complete [11481.578741] LustreError: 340476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.597805] LustreError: 340476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11487.144521] LustreError: 340462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593407 with bad export cookie 2641977891307296406 [11487.177524] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11487.183291] LustreError: 340462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11488.054227] Lustre: server umount lustre-MDT0001 complete [11503.211571] Lustre: server umount lustre-OST0000 complete [11518.176161] Lustre: server umount lustre-OST0001 complete [11522.567398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11524.331151] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11526.275709] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 03:30:45 (1788593445) [11552.577519] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11562.928167] loop0: detected capacity change from 0 to 1099511627776 [11563.730841] SGI XFS with ACLs, security attributes, quota, no debug enabled [11563.745162] *** VALIDATE xfs *** [11563.764915] XFS (loop0): Mounting V5 Filesystem [11563.782874] XFS (loop0): Ending clean mount [11565.243160] loop1: detected capacity change from 0 to 18691697672192 [11687.815821] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [11698.343836] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [11734.380997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.631665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11734.656943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11734.708484] Lustre: lustre-MDT0000: new disk, initializing [11734.748674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11734.759895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11739.772197] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11751.215266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.376812] Lustre: 348635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11751.436509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11751.441419] Lustre: Skipped 1 previous similar message [11751.530500] Lustre: lustre-MDT0001: new disk, initializing [11751.672425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11751.689556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11757.845209] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11769.594592] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11776.541612] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11784.068658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.404585] Lustre: lustre-OST0000: new disk, initializing [11784.411759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11784.434292] Lustre: 350607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11785.516096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11785.531868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11785.651740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11792.306455] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11802.005958] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11807.932946] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11808.384399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11813.817026] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11814.092868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11816.417562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11816.419625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.423290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11816.427073] Lustre: Skipped 4 previous similar messages [11816.443222] Lustre: Skipped 5 previous similar messages [11821.537654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11821.549891] Lustre: Skipped 1 previous similar message [11830.239124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11830.564364] Lustre: server umount lustre-OST0000 complete [11836.906664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.934543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.944120] Lustre: Skipped 3 previous similar messages [11840.375655] Lustre: server umount lustre-MDT0000 complete [11842.030951] LustreError: 348647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.041673] LustreError: 348647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11843.891169] LustreError: 348626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593764 with bad export cookie 2641977891307297953 [11843.892576] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11843.897662] LustreError: 348626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11844.322935] Lustre: server umount lustre-MDT0001 complete [11853.871750] XFS (loop0): Unmounting Filesystem [11858.953094] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 03:36:18 (1788593778) [11866.250315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.768860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11866.771678] Lustre: Skipped 2 previous similar messages [11871.415287] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11880.540510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.871691] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11893.603760] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11898.952294] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11905.734850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11912.582538] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [11921.940598] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11935.714864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11935.735593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11935.744638] Lustre: Skipped 2 previous similar messages [11941.582668] Lustre: server umount lustre-OST0000 complete [11951.741670] Lustre: server umount lustre-MDT0000 complete [11955.949027] LustreError: 353148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593876 with bad export cookie 2641977891307298835 [11955.955641] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11955.959616] LustreError: 353148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11956.435490] Lustre: server umount lustre-MDT0001 complete [11967.469376] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [11970.461942] Key type lgssc unregistered [11970.798897] LNet: 356795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11970.808938] LNetError: 356795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11971.881861] LNet: Removed LNI 192.168.202.151@tcp [11972.925132] Key type .llcrypt unregistered [11972.932044] Key type ._llcrypt unregistered [12000.364473] Key type ._llcrypt registered [12000.372526] Key type .llcrypt registered [12000.483327] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12001.990939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12002.045853] alg: No test for adler32 (adler32-zlib) [12003.246860] Lustre: Lustre: Build Version: 2.17.57_104_g4a8f637 [12003.464991] LNet: Added LNI 192.168.202.151@tcp [8/256/0/180] [12005.135206] Key type lgssc registered [12006.306985] Lustre: Echo OBD driver; http://www.lustre.org/ [12015.098542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12015.110970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.709439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12022.349435] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12026.858872] LustreError: 358360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.895500] LustreError: 358360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12031.974569] LustreError: 358361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12032.404909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.818474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12037.671812] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12045.955803] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12052.998701] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12060.742042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12061.348298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12068.106188] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12076.652667] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12088.718238] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.51@tcp (stopping) [12089.312170] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12089.329132] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.342550] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12089.372091] Lustre: Skipped 1 previous similar message [12089.391233] Lustre: Skipped 1 previous similar message [12092.772363] Lustre: server umount lustre-MDT0001 complete [12093.820929] LustreError: 358355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.933801] LustreError: 358356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.962682] LustreError: 358356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12104.056937] LustreError: 358356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12104.081428] LustreError: 358356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12106.747122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12106.949108] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12106.963687] Lustre: Found index 1 for lustre-MDT0001, updating log [12107.024075] Lustre: 358349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12107.523260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12112.882375] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12113.236389] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12115.963511] LustreError: 361936:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12115.966150] LustreError: 361936:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12115.979766] LustreError: 358352:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12116.089614] LustreError: 361943:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -17 [12116.106108] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12116.106108] [12123.104515] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12123.108121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12123.115975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12123.125341] Lustre: Skipped 1 previous similar message [12123.136869] Lustre: Skipped 1 previous similar message [12127.776673] Lustre: server umount lustre-MDT0001 complete [12128.233373] LustreError: 358360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12128.242862] LustreError: 358360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12140.056950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.215251] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12140.233922] Lustre: Found index 1 for lustre-MDT0001, updating log [12140.295665] Lustre: 358348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12140.550486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12145.674429] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12145.719711] LustreError: 362833:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12145.731155] LustreError: 362833:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12145.739114] LustreError: 358352:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12145.745062] LustreError: 358352:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12145.880836] LustreError: 362841:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12145.894844] LustreError: 362841:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12145.910426] LustreError: 362841:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -17 [12145.937796] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.151@tcp [12145.937796] [12146.933987] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12455.906983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12455.940686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12455.953745] Lustre: Skipped 1 previous similar message [12459.936200] Lustre: server umount lustre-MDT0001 complete [12461.028598] LustreError: 358360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.058293] LustreError: 358360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12472.280747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.362664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12472.367314] Lustre: Found index 1 for lustre-MDT0001, updating log [12472.401420] Lustre: 358348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12472.712792] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12476.038263] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12477.948924] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12480.513074] LustreError: 371325:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12480.522720] LustreError: 371325:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12480.540061] LustreError: 358352:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12480.549198] LustreError: 358352:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12480.662353] LustreError: 371328:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12480.682514] LustreError: 371328:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.151@tcp: cfg command failed: rc = -114 [12480.688545] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12480.688545] [12800.491468] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.503272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12800.512490] Lustre: Skipped 1 previous similar message [12800.514612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12800.514619] Lustre: Skipped 1 previous similar message [12804.299147] Lustre: server umount lustre-MDT0000 complete [12805.610352] LustreError: 358355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12805.630587] LustreError: 358355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12809.540676] LustreError: 358340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594730 with bad export cookie 2334401387424498809 [12809.565400] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12809.577624] LustreError: 358340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12810.722385] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12810.747601] Lustre: Skipped 2 previous similar messages [12810.762934] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12810.772437] Lustre: Skipped 2 previous similar messages [12816.599457] Lustre: server umount lustre-MDT0001 complete [12823.020962] Lustre: server umount lustre-OST0000 complete [12833.651481] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [12845.504447] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [12886.589650] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 03:53:25 (1788594805) [12894.057031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12894.371458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12894.392111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12894.501916] Lustre: lustre-MDT0000: new disk, initializing [12894.609432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12894.633771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12899.958568] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12910.650907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.947910] Lustre: 381948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12911.077360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12911.080782] Lustre: Skipped 1 previous similar message [12911.208602] Lustre: lustre-MDT0001: new disk, initializing [12911.349959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12911.401091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12911.411777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12916.760253] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12927.217769] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12933.967170] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12941.195805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12941.515208] Lustre: lustre-OST0000: new disk, initializing [12941.518928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12941.525776] Lustre: 383916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12941.605031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12948.007455] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_default_debug -1 all [12948.575792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12948.586513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12948.714346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12958.470764] Lustre: DEBUG MARKER: oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12967.926574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12967.931018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12967.935527] LustreError: Skipped 1 previous similar message [12967.966071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12967.973662] Lustre: Skipped 1 previous similar message [12973.212432] Lustre: server umount lustre-MDT0000 complete [12974.571406] LustreError: 383172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12974.590404] LustreError: 383172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12977.235267] LustreError: 381940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594898 with bad export cookie 2334401387424500762 [12977.242528] LustreError: MGC192.168.202.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12977.250138] LustreError: 381940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12977.250814] LustreError: 382753:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12977.251178] LustreError: 382753:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000166ba506 x1875476662766848/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12977.925912] Lustre: server umount lustre-MDT0001 complete [12993.140745] Lustre: server umount lustre-OST0000 complete [13024.316762] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing set_hostid [13036.034604] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing load_modules_local [13070.014449] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12789 sec ========= 03:56:29 (1788594989) [13072.367554] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:56:31 (1788594991) === [13076.371614] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:56:35 (1788594995) === [13122.226604] Lustre: DEBUG MARKER: oleg251-server.virtnet: executing unload_modules_local [13126.038461] Key type lgssc unregistered [13126.544875] LNet: 390284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13126.556972] LNetError: 390284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13126.592752] LNet: Removed LNI 192.168.202.151@tcp [13127.860845] Key type .llcrypt unregistered [13127.864363] Key type ._llcrypt unregistered