[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460752136 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002189] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010039] Yama: becoming mindful. [ 0.010733] SELinux: Initializing. [ 0.011047] *** VALIDATE selinux *** [ 0.018363] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022653] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024081] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025090] *** VALIDATE tmpfs *** [ 0.026348] *** VALIDATE proc *** [ 0.027175] *** VALIDATE cgroup *** [ 0.028005] *** VALIDATE cgroup2 *** [ 0.029248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032031] Spectre V2 : User space: Vulnerable [ 0.033010] Speculative Store Bypass: Vulnerable [ 0.036382] debug: unmapping init [mem 0xffffffff86259000-0xffffffff86260fff] [ 0.039192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041022] ... version: 2 [ 0.042014] ... bit width: 48 [ 0.043022] ... generic registers: 4 [ 0.043784] ... value mask: 0000ffffffffffff [ 0.044014] ... max period: 00007fffffffffff [ 0.045011] ... fixed-purpose events: 3 [ 0.046014] ... event mask: 000000070000000f [ 0.047324] rcu: Hierarchical SRCU implementation. [ 0.049482] smp: Bringing up secondary CPUs ... [ 0.050599] x86: Booting SMP configuration: [ 0.051026] .... node #0, CPUs: #1 #2 #3 [ 0.054053] smp: Brought up 1 node, 4 CPUs [ 0.056011] smpboot: Max logical packages: 1 [ 0.057015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139037] node 0 deferred pages initialised in 80ms [ 0.142119] devtmpfs: initialized [ 0.144292] x86/mm: Memory block size: 128MB [ 0.147463] gcov: version magic: 0x41383552 [ 0.150165] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153156] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156245] pinctrl core: initialized pinctrl subsystem [ 0.158182] [ 0.158742] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164011] ** ** [ 0.166011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169013] ** ** [ 0.171010] ** This means that this kernel is built to expose internal ** [ 0.173014] ** IOMMU data structures, which may compromise security on ** [ 0.176013] ** your system. ** [ 0.178017] ** ** [ 0.180010] ** If you see this message and you are not debugging the ** [ 0.182013] ** kernel, report this immediately to your vendor! ** [ 0.184014] ** ** [ 0.186012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189011] ************************************************************* [ 0.191867] NET: Registered protocol family 16 [ 0.193435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204135] cpuidle: using governor menu [ 0.206588] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207304] PCI: Using configuration type 1 for base access [ 0.209117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215141] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217152] cryptd: max_cpu_qlen set to 1000 [ 0.220776] ACPI: Added _OSI(Module Device) [ 0.222071] ACPI: Added _OSI(Processor Device) [ 0.223015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.231447] ACPI: Interpreter enabled [ 0.232098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234026] ACPI: Using IOAPIC for interrupt routing [ 0.235097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246543] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.252079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255956] acpiphp: Slot [2] registered [ 0.257067] acpiphp: Slot [5] registered [ 0.258080] acpiphp: Slot [6] registered [ 0.258824] acpiphp: Slot [7] registered [ 0.259059] acpiphp: Slot [8] registered [ 0.259918] acpiphp: Slot [9] registered [ 0.261081] acpiphp: Slot [10] registered [ 0.262085] acpiphp: Slot [3] registered [ 0.262819] acpiphp: Slot [4] registered [ 0.264092] acpiphp: Slot [11] registered [ 0.265120] acpiphp: Slot [12] registered [ 0.266105] acpiphp: Slot [13] registered [ 0.266951] acpiphp: Slot [14] registered [ 0.268069] acpiphp: Slot [15] registered [ 0.268892] acpiphp: Slot [16] registered [ 0.270162] acpiphp: Slot [17] registered [ 0.272099] acpiphp: Slot [18] registered [ 0.273037] acpiphp: Slot [19] registered [ 0.274092] acpiphp: Slot [20] registered [ 0.275103] acpiphp: Slot [21] registered [ 0.277100] acpiphp: Slot [22] registered [ 0.278136] acpiphp: Slot [23] registered [ 0.279072] acpiphp: Slot [24] registered [ 0.279971] acpiphp: Slot [25] registered [ 0.281103] acpiphp: Slot [26] registered [ 0.283031] acpiphp: Slot [27] registered [ 0.284094] acpiphp: Slot [28] registered [ 0.286069] acpiphp: Slot [29] registered [ 0.287111] acpiphp: Slot [30] registered [ 0.289115] acpiphp: Slot [31] registered [ 0.291091] PCI host bridge to bus 0000:00 [ 0.293022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.296024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.312049] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315095] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328429] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338573] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346769] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374584] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.405018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415325] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.427014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453849] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.460016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.468023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.483017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.493156] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.507019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.521019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.528941] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.536016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.542012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.560025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.570838] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.577017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.586018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.600018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.611328] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.614342] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.618346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.620239] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.622241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.629040] iommu: Default domain type: Passthrough [ 0.630504] SCSI subsystem initialized [ 0.632222] ACPI: bus type USB registered [ 0.633071] usbcore: registered new interface driver usbfs [ 0.635164] usbcore: registered new interface driver hub [ 0.637089] usbcore: registered new device driver usb [ 0.639182] pps_core: LinuxPPS API ver. 1 registered [ 0.641013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.644063] PTP clock support registered [ 0.647048] EDAC MC: Ver: 3.0.0 [ 0.649148] PCI: Using ACPI for IRQ routing [ 0.650905] NetLabel: Initializing [ 0.652011] NetLabel: domain hash size = 128 [ 0.654011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.656083] NetLabel: unlabeled traffic allowed by default [ 0.658122] vgaarb: loaded [ 0.659259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.661015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.667319] clocksource: Switched to clocksource kvm-clock [ 0.767127] VFS: Disk quotas dquot_6.6.0 [ 0.769914] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.773487] *** VALIDATE ramfs *** [ 0.775261] *** VALIDATE hugetlbfs *** [ 0.777344] pnp: PnP ACPI init [ 0.780169] pnp: PnP ACPI: found 6 devices [ 0.797589] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.802524] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.804276] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.806789] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.809021] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.811058] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.813131] NET: Registered protocol family 2 [ 0.815090] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.820207] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.823945] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.828734] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.832466] TCP: Hash tables configured (established 65536 bind 65536) [ 0.834850] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.837729] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.839823] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.841967] NET: Registered protocol family 1 [ 0.844523] RPC: Registered named UNIX socket transport module. [ 0.845790] RPC: Registered udp transport module. [ 0.847010] RPC: Registered tcp transport module. [ 0.848583] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.850739] NET: Registered protocol family 44 [ 0.852240] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.854120] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.856358] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.858219] PCI: CLS 0 bytes, default 64 [ 0.859207] Unpacking initramfs... [ 2.162777] debug: unmapping init [mem 0xffff981e3cc54000-0xffff981e3ffbffff] [ 2.165170] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.166447] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.167991] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.651828] Initialise system trusted keyrings [ 2.653581] Key type blacklist registered [ 2.655563] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.665240] zbud: loaded [ 2.668314] *** VALIDATE nfs *** [ 2.669660] *** VALIDATE nfs4 *** [ 2.671336] pstore: using deflate compression [ 2.674581] Platform Keyring initialized [ 2.764321] NET: Registered protocol family 38 [ 2.766219] Key type asymmetric registered [ 2.768041] Asymmetric key parser 'x509' registered [ 2.770165] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.773189] io scheduler mq-deadline registered [ 2.774351] io scheduler kyber registered [ 2.775535] io scheduler bfq registered [ 2.777337] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.780470] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.783405] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.786346] ACPI: Power Button [PWRF] [ 2.792102] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.798671] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.811322] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.816538] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.829135] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.856122] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.886911] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.892666] Non-volatile memory driver v1.3 [ 2.894142] Linux agpgart interface v0.103 [ 2.923958] virtio_blk virtio1: [vda] 149456 512-byte logical blocks (76.5 MB/73.0 MiB) [ 2.927048] vda: detected capacity change from 0 to 76521472 [ 2.941574] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.944804] vdb: detected capacity change from 0 to 1073741824 [ 2.960511] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.963785] vdc: detected capacity change from 0 to 2621440000 [ 2.982674] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.984978] vdd: detected capacity change from 0 to 2621440000 [ 2.999823] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.002860] vde: detected capacity change from 0 to 4294967296 [ 3.018577] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.021187] vdf: detected capacity change from 0 to 4294967296 [ 3.032742] libphy: Fixed MDIO Bus: probed [ 3.054200] usbcore: registered new interface driver usbserial_generic [ 3.057538] usbserial: USB Serial support registered for generic [ 3.060549] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.065095] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.067038] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.069676] mousedev: PS/2 mouse device common for all mice [ 3.073567] rtc_cmos 00:05: RTC can wake from S4 [ 3.075227] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.076823] rtc_cmos 00:05: registered as rtc0 [ 3.079896] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.081686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.083610] intel_pstate: CPU model not supported [ 3.088294] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.091881] hid: raw HID events driver (C) Jiri Kosina [ 3.094234] usbcore: registered new interface driver usbhid [ 3.096415] usbhid: USB HID core driver [ 3.098473] drop_monitor: Initializing network drop monitor service [ 3.101165] Initializing XFRM netlink socket [ 3.103313] NET: Registered protocol family 10 [ 3.106408] Segment Routing with IPv6 [ 3.107553] NET: Registered protocol family 17 [ 3.109605] mpls_gso: MPLS GSO support [ 3.116148] RAS: Correctable Errors collector initialized. [ 3.118492] AVX version of gcm_enc/dec engaged. [ 3.120439] AES CTR mode by8 optimization enabled [ 3.198249] sched_clock: Marking stable (3198208502, 0)->(3993020088, -794811586) [ 3.201762] registered taskstats version 1 [ 3.203776] Loading compiled-in X.509 certificates [ 3.205817] zswap: loaded using pool lzo/zbud [ 3.232598] Key type big_key registered [ 3.245979] Key type encrypted registered [ 3.247803] ima: No TPM chip found, activating TPM-bypass! [ 3.250020] ima: Allocated hash algorithm: sha1 [ 3.251823] ima: No architecture policies found [ 3.253634] evm: Initialising EVM extended attributes: [ 3.255615] evm: security.selinux [ 3.256572] evm: security.ima [ 3.257568] evm: security.capability [ 3.258970] evm: HMAC attrs: 0x1 [ 3.262778] rtc_cmos 00:05: setting system clock to 2026-08-22 07:15:02 UTC (1787382902) [ 3.269406] debug: unmapping init [mem 0xffffffff87203000-0xffffffff873fffff] [ 3.272549] debug: unmapping init [mem 0xffffffff85f82000-0xffffffff86258fff] [ 3.281222] Write protecting the kernel read-only data: 28672k [ 3.284857] debug: unmapping init [mem 0xffffffff84603000-0xffffffff847fffff] [ 3.288629] debug: unmapping init [mem 0xffffffff84f14000-0xffffffff84ffffff] [ 3.327593] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.336764] systemd[1]: Detected virtualization kvm. [ 3.338840] systemd[1]: Detected architecture x86-64. [ 3.340944] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.370528] systemd[1]: No hostname configured. [ 3.372508] systemd[1]: Set hostname to . [ 3.374872] random: systemd: uninitialized urandom read (16 bytes read) [ 3.377660] systemd[1]: Initializing machine ID from random generator. [ 3.424603] random: ln: uninitialized urandom read (6 bytes read) [ 3.524691] random: systemd: uninitialized urandom read (16 bytes read) [ 3.526790] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.531513] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.534980] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.199291] device-mapper: uevent: version 1.0.3 [ 4.201559] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.040520] virtio_net virtio0 ens2: renamed from eth0 [ 5.064676] random: fast init done [ 5.103961] scsi host0: ata_piix [ 5.123255] scsi host1: ata_piix [ 5.125313] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.127926] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.744968] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.772719] random: crng init done [ 9.774137] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.310174] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.445723] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.701031] SELinux: Disabled at runtime. [ 11.764510] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.770714] systemd[1]: Detected virtualization kvm. [ 11.772054] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.229377] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.231746] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.235249] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.237507] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.240289] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.248151] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.256685] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped targ[ 12.369106] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Initrd File Systems. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Col[ 12.697343] squashfs: version 4.0 (2009/01/31) Phillip Lougher dplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.961768] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.977982] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.148783] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.156122] EDAC sbridge: Ver: 1.1.2 [ 14.731254] Key type dns_resolver registered [ 15.050210] NFS: Registering the id_resolver key type [ 15.052274] Key type id_resolver registered [ 15.053790] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-server login: [ 45.709430] libcfs: loading out-of-tree module taints kernel. [ 45.761730] Key type ._llcrypt registered [ 45.764902] Key type .llcrypt registered [ 45.940208] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 68.583542] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 70.699640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.711115] alg: No test for adler32 (adler32-zlib) [ 72.063463] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 73.313292] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 75.080321] Key type lgssc registered [ 76.774270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.075521] hrtimer: interrupt took 19190786 ns [ 101.554139] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.166580] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 154.250172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.302429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.575812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.625110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.708930] Lustre: lustre-MDT0000: new disk, initializing [ 155.833558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.860831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.312885] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.336740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.505213] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.545113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.551069] Lustre: Skipped 1 previous similar message [ 172.611528] Lustre: lustre-MDT0001: new disk, initializing [ 172.666198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.693295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.709944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.149972] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.866255] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.299292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.579512] Lustre: lustre-OST0000: new disk, initializing [ 191.584956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.593147] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.657907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.889945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.902643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.977473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.624279] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.434943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.566636] Lustre: lustre-OST0001: new disk, initializing [ 211.568924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.582570] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.650506] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.302281] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.740345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.752221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.903303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.552203] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.824292] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 240.834427] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.623702] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing yml_node [ 249.379374] Lustre: DEBUG MARKER: Client: 2.17.54.114 [ 251.400351] Lustre: DEBUG MARKER: MDS: 2.17.54.114 [ 253.481494] Lustre: DEBUG MARKER: OSS: 2.17.54.114 [ 254.836213] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:19:12 EDT 2026 [ 270.004509] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 271.486293] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.787333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.800131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.815068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.809826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.811196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.823051] Lustre: Skipped 1 previous similar message [ 291.826778] Lustre: Skipped 1 previous similar message [ 294.312816] Lustre: server umount lustre-MDT0000 complete [ 298.144157] LustreError: 7459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383197 with bad export cookie 5129387231821477030 [ 298.147725] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.152112] LustreError: 7459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 302.048652] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.057669] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.068185] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.068196] Lustre: Skipped 2 previous similar messages [ 302.071795] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 302.114525] Lustre: Skipped 2 previous similar messages [ 303.074382] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.089634] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 304.524196] Lustre: server umount lustre-MDT0001 complete [ 308.891572] Lustre: server umount lustre-OST0000 complete [ 312.782589] Lustre: server umount lustre-OST0001 complete [ 320.055598] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 328.799282] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 363.763608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.948173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 363.973886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.036465] Lustre: lustre-MDT0000: new disk, initializing [ 364.095146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.114458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.016483] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 378.488742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.589841] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.641489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.645482] Lustre: Skipped 1 previous similar message [ 378.782171] Lustre: lustre-MDT0001: new disk, initializing [ 378.865454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.900444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 378.910154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 383.882858] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 393.225381] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 400.127779] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 407.571705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.069959] Lustre: lustre-OST0000: new disk, initializing [ 408.087847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.099059] Lustre: 17039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 408.224239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.622882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.635342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.778646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 414.777825] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 423.493622] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 428.037654] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 428.266439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 432.614363] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 432.857343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 435.169631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 435.178077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.193871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.200839] Lustre: Skipped 3 previous similar messages [ 440.817772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.829693] Lustre: Skipped 1 previous similar message [ 441.029436] Lustre: server umount lustre-OST0000 complete [ 445.420105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 445.433841] LustreError: Skipped 1 previous similar message [ 445.438731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.456872] Lustre: Skipped 1 previous similar message [ 450.914352] Lustre: server umount lustre-MDT0000 complete [ 451.043905] LustreError: 15944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.068395] LustreError: 15944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 454.500280] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383353 with bad export cookie 5129387231821478570 [ 454.503774] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.511798] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 454.931456] Lustre: server umount lustre-MDT0001 complete [ 460.419619] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 03:22:37 (1787383357) [ 466.947590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.609431] LustreError: 19302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.688610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.527891] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 473.067399] LustreError: 19303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.460708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.816210] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 492.455683] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 497.913868] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 504.176621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 510.097405] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 518.054144] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 521.962207] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 527.841565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 527.848487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.875332] Lustre: Skipped 1 previous similar message [ 527.886065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.893877] Lustre: Skipped 3 previous similar messages [ 534.136354] Lustre: server umount lustre-MDT0000 complete [ 537.871677] LustreError: 19283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383437 with bad export cookie 5129387231821479452 [ 537.893582] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.305518] Lustre: server umount lustre-MDT0001 complete [ 552.695309] Lustre: server umount lustre-OST0000 complete [ 575.336615] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 586.111298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.457227] LustreError: 23695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.470565] LustreError: 23695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 586.540827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.546948] Lustre: Skipped 2 previous similar messages [ 591.303094] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 599.123195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.342830] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 606.450599] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 612.450322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 618.702396] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 627.757396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 627.982910] Lustre: lustre-OST0001: new disk, initializing [ 627.996935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 628.010801] Lustre: 26008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 629.628990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 629.636333] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 629.734465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 634.623725] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 643.788812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 647.034874] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 660.970380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 660.983905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.999783] Lustre: Skipped 2 previous similar messages [ 661.003385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 661.008348] Lustre: Skipped 6 previous similar messages [ 665.083935] Lustre: server umount lustre-MDT0000 complete [ 667.624984] LustreError: 23691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.662797] LustreError: 23691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 669.538638] LustreError: 23677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383568 with bad export cookie 5129387231821480299 [ 669.541653] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.554238] LustreError: 23677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 670.007575] Lustre: server umount lustre-MDT0001 complete [ 684.342084] Lustre: server umount lustre-OST0000 complete [ 698.442635] Lustre: server umount lustre-OST0001 complete [ 713.833907] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 723.501788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.755129] LustreError: 29027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.825805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.829406] Lustre: Skipped 3 previous similar messages [ 727.856857] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 736.764117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.749691] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 747.270768] Lustre: 30175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 755.433756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 764.195650] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 773.197531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 780.294983] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 787.581415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 790.790950] Lustre: 32053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 798.691882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 798.707441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 798.732716] Lustre: Skipped 3 previous similar messages [ 798.749637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.764397] Lustre: Skipped 4 previous similar messages [ 802.957987] Lustre: server umount lustre-MDT0000 complete [ 804.345425] LustreError: 29027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.370758] LustreError: 29027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 807.527080] LustreError: 29009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383706 with bad export cookie 5129387231821481587 [ 807.536471] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.546335] LustreError: 29009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 808.454887] Lustre: server umount lustre-MDT0001 complete [ 823.656984] Lustre: server umount lustre-OST0000 complete [ 825.632179] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383708/real 1787383708] req@ffff981d83508700 x1874206887532928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 827.875877] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787383711/real 1787383711] req@ffff981eb7e43b80 x1874206887533184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787383727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 828.401921] Lustre: server umount lustre-OST0001 complete [ 837.202986] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 03:28:54 (1787383734) [ 865.850318] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 876.212714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.942411] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 890.857541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.272588] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 899.012187] Lustre: 36001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 905.870679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.345648] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 920.760312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.885920] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 934.048827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 937.913198] Lustre: 37881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 947.682512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.692216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 947.696110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.696120] Lustre: Skipped 4 previous similar messages [ 947.705205] Lustre: Skipped 7 previous similar messages [ 949.614167] Lustre: server umount lustre-MDT0000 complete [ 952.805962] LustreError: 37852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.816159] LustreError: 37852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 953.151821] LustreError: 34835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383852 with bad export cookie 5129387231821482700 [ 953.163632] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.169116] LustreError: 34835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 953.441198] Lustre: server umount lustre-MDT0001 complete [ 966.240719] Lustre: server umount lustre-OST0000 complete [ 980.371767] Lustre: server umount lustre-OST0001 complete [ 989.986811] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 03:31:27 (1787383887) [ 1007.118796] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1017.719914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.530472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.536139] Lustre: Skipped 7 previous similar messages [ 1023.651120] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1032.827671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.472201] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1040.830295] Lustre: 41175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1046.512330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.510767] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1060.834161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.021869] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1076.801708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1080.244221] Lustre: 43054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1081.159486] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1081.164947] Lustre: Skipped 1 previous similar message [ 1084.386801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1084.394719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.414973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.419482] Lustre: Skipped 3 previous similar messages [ 1090.693788] Lustre: server umount lustre-MDT0000 complete [ 1094.045828] LustreError: 41177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383993 with bad export cookie 5129387231821483806 [ 1094.054090] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.683350] Lustre: server umount lustre-MDT0001 complete [ 1109.331731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.707684] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1129.464119] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1134.630265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.100350] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1140.730972] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1140.738936] Lustre: Skipped 1 previous similar message [ 1147.838838] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1149.826052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1156.168388] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1157.900501] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1165.808138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.590581] Lustre: server umount lustre-MDT0000 complete [ 1174.325497] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384073 with bad export cookie 5129387231821484933 [ 1174.336987] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.344798] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1174.676643] Lustre: server umount lustre-MDT0001 complete [ 1188.072073] Lustre: server umount lustre-OST0000 complete [ 1192.416248] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787384075/real 1787384075] req@ffff981d8373df80 x1874206887668224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787384091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1192.738878] Lustre: server umount lustre-OST0001 complete [ 1202.721106] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 03:34:59 (1787384099) [ 1218.934787] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1229.974030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.370500] LustreError: 47659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.383849] LustreError: 47659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 1234.996738] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1244.040908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.936608] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1251.923823] Lustre: 48812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1259.252909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.254068] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1275.103573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1282.455042] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1289.853982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1294.284991] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1306.082166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1308.231184] Lustre: server umount lustre-MDT0000 complete [ 1311.693627] LustreError: 47641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384210 with bad export cookie 5129387231821485724 [ 1311.696880] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.704199] LustreError: 47641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1312.105033] Lustre: server umount lustre-MDT0001 complete [ 1321.902698] Lustre: server umount lustre-OST0000 complete [ 1325.656889] Lustre: server umount lustre-OST0001 complete [ 1335.660642] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 03:37:13 (1787384233) [ 1338.223451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1340.230170] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 03:37:17 (1787384237) [ 1352.004856] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1360.134347] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1383.557380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.891134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1384.054603] Lustre: lustre-MDT0000: new disk, initializing [ 1384.146203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1389.004909] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1399.651572] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1407.209885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1407.586111] Lustre: lustre-OST0000: new disk, initializing [ 1407.590504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1407.596312] Lustre: Skipped 1 previous similar message [ 1407.612947] Lustre: 54885:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1408.874695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1408.880966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1408.930297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1413.896173] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1424.695558] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1427.606656] Lustre: DEBUG MARKER: create test files [ 1460.194799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.211918] Lustre: Skipped 16 previous similar messages [ 1460.221702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1460.231180] Lustre: Skipped 21 previous similar messages [ 1465.657042] Lustre: server umount lustre-OST0000 complete [ 1470.144635] Lustre: server umount lustre-MDT0000 complete [ 1481.306273] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1484.332847] Key type lgssc unregistered [ 1484.803519] LNet: 56516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.818050] LNetError: 56516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.845621] LNet: Removed LNI 192.168.202.153@tcp [ 1486.001286] Key type .llcrypt unregistered [ 1486.006979] Key type ._llcrypt unregistered [ 1514.872040] Key type ._llcrypt registered [ 1514.875407] Key type .llcrypt registered [ 1515.017704] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1517.035000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1517.186241] alg: No test for adler32 (adler32-zlib) [ 1518.491457] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1518.835977] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1520.616395] Key type lgssc registered [ 1521.860255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1531.240983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1531.254921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.671041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.050689] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1542.991073] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1550.782641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1551.290424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.449922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:98 to 0x240000400:129) [ 1557.585681] Lustre: 59246:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1557.591437] Lustre: 59246:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1557.595482] Lustre: 59246:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1557.599888] Lustre: 59246:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1557.611753] Lustre: 59246:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1557.620019] Lustre: 59246:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1558.571770] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1565.642990] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1571.512907] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1576.569127] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1583.073677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.089083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.101050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1588.228522] Lustre: server umount lustre-OST0000 complete [ 1595.153840] Lustre: server umount lustre-MDT0000 complete [ 1609.916398] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1613.104248] Key type lgssc unregistered [ 1613.522875] LNet: 60856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.533950] LNetError: 60856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.572229] LNet: Removed LNI 192.168.202.153@tcp [ 1615.634389] Key type .llcrypt unregistered [ 1615.637137] Key type ._llcrypt unregistered [ 1647.605665] Key type ._llcrypt registered [ 1647.607747] Key type .llcrypt registered [ 1647.680092] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1649.296893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.363973] alg: No test for adler32 (adler32-zlib) [ 1650.404935] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1650.718555] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1652.528224] Key type lgssc registered [ 1653.658986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1662.617981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1662.636332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.161904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.501095] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1678.185662] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1685.744368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.268679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.326863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1689.423238] Lustre: 63695:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 1689.430896] Lustre: 63695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/1, destroy: 0/0/0 [ 1689.438976] Lustre: 63695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1689.446528] Lustre: 63695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1689.456973] Lustre: 63695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1689.463623] Lustre: 63695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1693.317953] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1701.462247] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.007968] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1722.648520] Lustre: server umount lustre-OST0000 complete [ 1740.444742] Lustre: server umount lustre-MDT0000 complete [ 1754.163405] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1757.334823] Key type lgssc unregistered [ 1757.595195] LNet: 65254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.604993] LNetError: 65254:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.614770] LNet: Removed LNI 192.168.202.153@tcp [ 1758.623492] Key type .llcrypt unregistered [ 1758.625954] Key type ._llcrypt unregistered [ 1778.070264] Key type ._llcrypt registered [ 1778.078279] Key type .llcrypt registered [ 1778.235824] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1795.425446] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1796.905888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1796.932256] alg: No test for adler32 (adler32-zlib) [ 1797.955407] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 1798.223406] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1799.914045] Key type lgssc registered [ 1802.020917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1843.838952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1843.871135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.349145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1845.384783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1845.512923] Lustre: lustre-MDT0000: new disk, initializing [ 1845.590827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.616894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1850.975293] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1862.710842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.808667] Lustre: 68410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1862.857473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1862.860811] Lustre: Skipped 1 previous similar message [ 1862.934968] Lustre: lustre-MDT0001: new disk, initializing [ 1863.115489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.210512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1863.230167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1868.310694] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1879.534678] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.889376] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1894.409336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.792226] Lustre: lustre-OST0000: new disk, initializing [ 1894.796462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1894.802180] Lustre: 70377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1894.895521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.172746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1901.196488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1901.393868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1901.873278] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1912.232229] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1918.091546] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1918.386539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1923.404477] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1923.638717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1927.138387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1927.139271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.157080] LustreError: Skipped 1 previous similar message [ 1927.178086] Lustre: Skipped 1 previous similar message [ 1927.185426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.029684] Lustre: server umount lustre-OST0000 complete [ 1937.377575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.393937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.401055] Lustre: Skipped 2 previous similar messages [ 1942.469426] Lustre: server umount lustre-MDT0000 complete [ 1946.714393] LustreError: 68401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384845 with bad export cookie 7385756377996635492 [ 1946.716553] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.728544] LustreError: 68401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1947.088575] Lustre: server umount lustre-MDT0001 complete [ 1958.507639] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 03:47:35 (1787384855) [ 1973.601931] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2017.833482] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 2029.944350] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2067.284082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.602677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2067.639465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2067.799061] Lustre: lustre-MDT0000: new disk, initializing [ 2067.936596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.964520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2073.231846] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2084.387914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.494292] Lustre: 76399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2084.535479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2084.538079] Lustre: Skipped 1 previous similar message [ 2084.611520] Lustre: lustre-MDT0001: new disk, initializing [ 2084.712212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.762808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2084.773583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2089.312523] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2098.726401] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2104.428175] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2112.225989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2112.435530] Lustre: lustre-OST0000: new disk, initializing [ 2112.439504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2112.444424] Lustre: 78367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2112.525338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.095809] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2122.851280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2122.872600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2122.997878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2128.919792] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2133.551850] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2133.780613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2138.910705] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2139.176731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2143.717573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2143.730033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2143.749288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.758929] Lustre: Skipped 2 previous similar messages [ 2143.777314] Lustre: Skipped 1 previous similar message [ 2147.115926] Lustre: server umount lustre-OST0000 complete [ 2151.394164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.406871] LustreError: Skipped 1 previous similar message [ 2151.412791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.428206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.569947] Lustre: server umount lustre-MDT0000 complete [ 2159.082018] LustreError: 76409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.109214] LustreError: 76409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2161.115222] LustreError: 76391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385060 with bad export cookie 7385756377996636374 [ 2161.125049] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2161.139721] LustreError: 76391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2161.569797] Lustre: server umount lustre-MDT0001 complete [ 2171.373857] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 03:51:08 (1787385068) [ 2178.419997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.868578] LustreError: 80781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.967631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.617648] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2184.164677] LustreError: 80781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.287082] LustreError: 80782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.187420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.142697] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2204.430247] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2210.480701] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2216.962726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2217.239224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.249068] Lustre: Skipped 1 previous similar message [ 2223.748909] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2231.807909] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2235.808212] Lustre: 83452:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 033bc067-533c-438c-b8a7-d142ba8bf83b at adminstrative request [ 2239.989212] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.999225] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2250.217916] Lustre: MGS: Client 033bc067-533c-438c-b8a7-d142ba8bf83b (at 0@lo) reconnecting [ 2250.230388] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x667f7a4b5c10b448 to 0x667f7a4b5c10b6e8 [ 2250.277361] Lustre: MGC192.168.202.153@tcp: Connection restored to 0@lo (at 0@lo) [ 2273.601725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.858554] Lustre: lustre-OST0001: new disk, initializing [ 2273.880760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2273.894750] Lustre: 83847:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2279.544739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2279.555146] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2279.659847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2280.190838] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2289.494988] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2295.270883] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2295.278119] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.284205] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2295.296967] Lustre: Skipped 2 previous similar messages [ 2295.303804] Lustre: Skipped 2 previous similar messages [ 2298.489650] Lustre: server umount lustre-OST0001 complete [ 2300.388696] LustreError: 82602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.416031] LustreError: 82602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2305.514235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.519256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2305.522578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.522587] Lustre: Skipped 1 previous similar message [ 2305.536877] Lustre: Skipped 1 previous similar message [ 2305.576326] LustreError: Skipped 1 previous similar message [ 2310.628825] LustreError: 82602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.648579] LustreError: 82602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2317.281487] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2317.663364] Lustre: server umount lustre-OST0000 complete [ 2325.475825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2325.485767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.531066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2325.534523] Lustre: Skipped 5 previous similar messages [ 2336.224143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2336.707046] Lustre: server umount lustre-MDT0000 complete [ 2341.045230] LustreError: 80762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385240 with bad export cookie 7385756377996637928 [ 2341.055772] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.457544] Lustre: server umount lustre-MDT0001 complete [ 2351.043449] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 03:54:08 (1787385248) [ 2352.625353] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2354.646214] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 03:54:11 (1787385251) [ 2356.334605] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2358.372522] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 03:54:15 (1787385255) [ 2360.383893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2362.392666] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 03:54:19 (1787385259) [ 2364.220891] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2366.330422] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 03:54:23 (1787385263) [ 2368.390732] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2375.034936] loop: module loaded [ 2375.065569] loop0: detected capacity change from 0 to 4294967296 [ 2375.137967] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2377.165562] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2380.483672] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2385.046766] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2416.127824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.166698] LustreError: 87973:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2416.171723] LustreError: 87973:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2416.175683] LustreError: 87973:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2416.181208] LustreError: 87973:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2435.348314] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 2444.869263] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2478.068723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.367773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2478.399157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2478.511276] Lustre: lustre-MDT0000: new disk, initializing [ 2478.608922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.613780] Lustre: Skipped 1 previous similar message [ 2478.629295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2482.808811] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2492.790991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.927497] Lustre: 90840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2493.170043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2498.459512] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2507.597936] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2513.083652] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2519.571744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.786559] Lustre: 92808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2525.740622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2525.870636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2526.174125] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2536.512727] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2540.769963] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2540.949341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2545.704261] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2545.923877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2551.777652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2551.778468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.802622] LustreError: Skipped 1 previous similar message [ 2551.807521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2551.807531] Lustre: Skipped 5 previous similar messages [ 2551.870029] Lustre: Skipped 2 previous similar messages [ 2554.144981] Lustre: server umount lustre-OST0000 complete [ 2564.218360] Lustre: server umount lustre-MDT0000 complete [ 2567.137547] LustreError: 90851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.164166] LustreError: 90851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2568.831238] LustreError: 91636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385468 with bad export cookie 7385756377996638544 [ 2568.841592] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.847664] LustreError: 91636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2569.398773] Lustre: server umount lustre-MDT0001 complete [ 2578.923438] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 03:57:56 (1787385476) [ 2586.028314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.104517] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2599.984256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.927186] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2611.643483] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2616.563496] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2622.800883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2623.258477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.271151] Lustre: Skipped 4 previous similar messages [ 2629.642837] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2636.485228] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2642.064943] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2642.249614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2647.274552] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2647.488304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2653.823554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.946551] Lustre: lustre-OST0001: new disk, initializing [ 2653.948366] Lustre: Skipped 2 previous similar messages [ 2653.950254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2653.952306] Lustre: Skipped 3 previous similar messages [ 2653.956130] Lustre: 98592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2660.277815] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2664.074374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2664.090185] Lustre: Skipped 2 previous similar messages [ 2664.106193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2664.263730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2671.305321] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2677.943187] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2678.272426] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2683.492273] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2683.797815] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2702.808389] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2725.644107] Lustre: Failing over lustre-MDT0000 [ 2725.870645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.874718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.887958] Lustre: Skipped 5 previous similar messages [ 2725.905509] Lustre: Skipped 7 previous similar messages [ 2726.169712] Lustre: server umount lustre-MDT0000 complete [ 2726.943257] LustreError: 99992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.955962] LustreError: 99992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2747.364122] Lustre: 66632:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385630/real 1787385630] req@000000003619639b x1874208696316544/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787385646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2747.417905] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2751.379676] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2751.381626] LDISKFS-fs (dm-0): recovery complete [ 2751.395764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.695829] LustreError: 66630:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001736887d x1874208696325632/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2759.090274] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2762.730519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2762.732836] Lustre: 95218:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2763.973152] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2764.274530] Lustre: 95223:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2764.286900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2764.293462] Lustre: 95223:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message [ 2764.388537] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2764.712156] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2765.035161] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2765.041259] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2765.376782] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2765.386157] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2765.400978] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2765.430036] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages [ 2766.081244] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2766.097347] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2766.440167] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2766.448198] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2767.410610] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2767.420795] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 5 previous similar messages [ 2768.386044] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2768.399441] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 6 previous similar messages [ 2768.720617] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2768.729164] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 6 previous similar messages [ 2771.593579] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2771.610315] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 12 previous similar messages [ 2772.568903] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2772.577129] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2772.896155] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2772.899945] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2779.869365] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2779.883452] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 25 previous similar messages [ 2780.849279] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2780.864498] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 25 previous similar messages [ 2781.193266] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2781.205440] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 25 previous similar messages [ 2796.177627] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2796.187976] Lustre: 101441:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 50 previous similar messages [ 2797.162292] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2797.186260] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 50 previous similar messages [ 2797.507119] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2797.523438] LustreError: 101441:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 50 previous similar messages [ 2822.280481] Lustre: 101441:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2822.296557] Lustre: 101441:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2822.320363] Lustre: 101441:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 097d490a-af9f-4059-9a79-f3abd862c73a@192.168.202.53@tcp [ 2822.334149] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2822.349297] Lustre: 101441:0:(ldlm_lib.c:1900:abort_req_replay_queue()) @@@ aborted: req@00000000af5512cd x1874208678286848/t0(8589934778) o35->097d490a-af9f-4059-9a79-f3abd862c73a@192.168.202.53@tcp:242/0 lens 392/0 e 0 to 0 dl 1787385732 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2822.404197] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2822.413157] Lustre: lustre-MDT0000: Denying connection for new client 097d490a-af9f-4059-9a79-f3abd862c73a (at 192.168.202.53@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2822.463579] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2822.510833] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2822.527978] Lustre: Skipped 2 previous similar messages [ 2822.530862] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2822.570440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2822.583031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2840.546967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.556203] LustreError: Skipped 1 previous similar message [ 2845.597460] Lustre: server umount lustre-OST0000 complete [ 2846.204358] LustreError: 97042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.242598] LustreError: 97042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 2856.448601] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2856.453247] Lustre: Skipped 5 previous similar messages [ 2856.797828] Lustre: server umount lustre-OST0001 complete [ 2863.597438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.630392] Lustre: Skipped 4 previous similar messages [ 2876.386096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2876.805326] Lustre: server umount lustre-MDT0000 complete [ 2882.226935] LustreError: 95204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385781 with bad export cookie 7385756377996689784 [ 2882.229824] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.235655] LustreError: 95204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2882.590224] Lustre: server umount lustre-MDT0001 complete [ 2893.901702] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 04:03:10 (1787385790) [ 2904.690848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.704832] Lustre: *** cfs_fail_loc=197, val=0*** [ 2920.549215] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2925.728585] LustreError: 103434:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 2930.962848] Lustre: *** cfs_fail_loc=197, val=0*** [ 2930.968227] Lustre: Skipped 65 previous similar messages [ 2951.584410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.594043] Lustre: Skipped 2 previous similar messages [ 2959.593524] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2969.483600] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2972.156558] Lustre: server umount lustre-OST0000 complete [ 2981.585783] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 04:04:38 (1787385878) [ 3003.906896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.550753] LustreError: 105057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.579748] LustreError: 105057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3010.196141] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3021.055253] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3025.359185] LustreError: 105088:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3025.374814] LustreError: 105088:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3025.387239] LustreError: 105088:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3025.813949] Lustre: server umount lustre-MDT0000 complete [ 3063.585435] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3075.071355] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3119.325834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.596304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3119.637608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3119.743659] Lustre: lustre-MDT0000: new disk, initializing [ 3119.820107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3125.293473] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3137.472650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.613157] Lustre: 109263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3137.934308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3142.884730] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3153.413062] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3159.752959] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3167.853430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.321749] Lustre: 111234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3174.605655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3175.211216] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3186.085560] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3192.107191] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3192.440371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3198.391604] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3198.693314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3205.615493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.619077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.620853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.620861] Lustre: Skipped 6 previous similar messages [ 3205.633771] LustreError: Skipped 1 previous similar message [ 3205.704323] Lustre: Skipped 2 previous similar messages [ 3207.398954] Lustre: server umount lustre-OST0000 complete [ 3218.269479] Lustre: server umount lustre-MDT0000 complete [ 3222.973321] LustreError: 109256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386122 with bad export cookie 7385756377996700438 [ 3222.986335] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.002972] LustreError: 109256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3223.341918] Lustre: server umount lustre-MDT0001 complete [ 3233.745650] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 04:08:51 (1787386131) [ 3245.559418] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3249.335616] Key type lgssc unregistered [ 3249.655671] LNet: 113721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3249.664329] LNetError: 113721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3249.687137] LNet: Removed LNI 192.168.202.153@tcp [ 3250.937655] Key type .llcrypt unregistered [ 3250.941657] Key type ._llcrypt unregistered [ 3270.921481] Key type ._llcrypt registered [ 3270.926183] Key type .llcrypt registered [ 3271.053063] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3295.091845] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3297.752443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3297.843810] alg: No test for adler32 (adler32-zlib) [ 3298.893203] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3299.128040] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3301.016252] Key type lgssc registered [ 3302.926140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3353.403273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3353.453160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.844385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3354.882894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3355.023946] Lustre: lustre-MDT0000: new disk, initializing [ 3355.176300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.201551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3360.981708] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3372.197324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.589866] Lustre: lustre-OST0000: new disk, initializing [ 3372.595672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3372.600624] Lustre: Skipped 1 previous similar message [ 3372.606658] Lustre: 117895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3372.756510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.981880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3380.998620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3381.090770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3381.947867] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3396.320491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3396.551342] Lustre: lustre-OST0001: new disk, initializing [ 3396.555267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3396.564633] Lustre: 118759:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3396.748982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3402.525638] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3405.004335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3405.036075] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3405.092672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3415.523316] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.531885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.538597] Lustre: Skipped 1 previous similar message [ 3418.651973] Lustre: server umount lustre-MDT0000 complete [ 3425.851754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3431.317334] LustreError: 117083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386330 with bad export cookie 12696836878440126996 [ 3431.325089] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.334098] LustreError: 117083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3431.473722] Lustre: server umount lustre-OST0000 complete [ 3435.997900] Lustre: server umount lustre-OST0001 complete [ 3446.228290] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 04:12:23 (1787386343) [ 3470.713457] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 04:12:47 (1787386367) [ 3482.692891] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3506.987648] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 04:13:24 (1787386404) [ 3549.723841] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3560.164281] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3597.908284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.289550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3598.329749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3598.410374] Lustre: lustre-MDT0000: new disk, initializing [ 3598.485849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.500362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3603.480265] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3614.113967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.270834] Lustre: 126516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3614.327630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3614.332381] Lustre: Skipped 1 previous similar message [ 3614.388434] Lustre: lustre-MDT0001: new disk, initializing [ 3614.481801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.512162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3614.522217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3619.460674] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3628.252938] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3633.696824] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3640.715991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3640.934950] Lustre: lustre-OST0000: new disk, initializing [ 3640.939462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3640.945490] Lustre: 128483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3641.028518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.008767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3647.021750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3647.153775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3647.406423] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3655.437419] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3659.837319] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3660.005128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3664.826718] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3665.070036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3672.551160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.557280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.574301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3672.584823] Lustre: Skipped 1 previous similar message [ 3673.068267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.357582] Lustre: server umount lustre-OST0000 complete [ 3681.249438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3681.262111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.279692] Lustre: Skipped 1 previous similar message [ 3681.293260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3681.295997] Lustre: Skipped 2 previous similar messages [ 3683.301058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.303094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.969069] Lustre: server umount lustre-MDT0000 complete [ 3689.063850] LustreError: 126510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386588 with bad export cookie 12696836878440128221 [ 3689.071689] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3689.078435] LustreError: 126510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3689.368861] Lustre: server umount lustre-MDT0001 complete [ 3697.529470] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 04:16:34 (1787386594) [ 3704.453582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.137870] LustreError: 130897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.165515] LustreError: 130897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3705.265673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.324138] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3710.448770] LustreError: 130898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.554531] LustreError: 130897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.493880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.957825] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3729.676660] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3735.281762] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3743.595939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.918418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.928518] Lustre: Skipped 1 previous similar message [ 3752.336091] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3761.350938] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3765.844342] Lustre: *** cfs_fail_loc=159, val=0*** [ 3781.160709] Lustre: lustre-MDT0000: Client 4f3b14d5-44a8-49c9-bffe-3acc2b704cbd (at 192.168.202.53@tcp) reconnecting [ 3783.010556] Lustre: *** cfs_fail_loc=159, val=0*** [ 3783.012542] Lustre: Skipped 5 previous similar messages [ 3798.579359] Lustre: lustre-MDT0000: Client 4f3b14d5-44a8-49c9-bffe-3acc2b704cbd (at 192.168.202.53@tcp) reconnecting [ 3816.930935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.936465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.939087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3816.939104] Lustre: Skipped 1 previous similar message [ 3816.942598] Lustre: Skipped 1 previous similar message [ 3822.596397] Lustre: server umount lustre-OST0000 complete [ 3826.657475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3826.666510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.686710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3826.692566] Lustre: Skipped 3 previous similar messages [ 3832.900325] Lustre: server umount lustre-MDT0000 complete [ 3836.355537] LustreError: 130879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386735 with bad export cookie 12696836878440129103 [ 3836.365448] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.378330] LustreError: 130879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3836.904563] Lustre: server umount lustre-MDT0001 complete [ 3848.316638] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3851.220551] Key type lgssc unregistered [ 3851.563081] LNet: 134674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3851.570440] LNetError: 134674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.587678] LNet: Removed LNI 192.168.202.153@tcp [ 3852.417615] Key type .llcrypt unregistered [ 3852.420136] Key type ._llcrypt unregistered [ 3878.979297] Key type ._llcrypt registered [ 3878.982237] Key type .llcrypt registered [ 3879.110344] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3880.441985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3880.653263] alg: No test for adler32 (adler32-zlib) [ 3881.848569] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 3882.056887] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3883.705721] Key type lgssc registered [ 3885.262720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3894.362747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3894.378660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.899969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.951660] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3906.024166] LustreError: 136090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.058025] LustreError: 136090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3911.143672] LustreError: 136091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.277782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.842758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3916.885528] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3925.207507] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3931.008316] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3937.808428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.183303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.242788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3944.463760] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3952.198491] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3966.312820] Lustre: *** cfs_fail_loc=159, val=0*** [ 3982.874875] Lustre: lustre-MDT0000: Client b656ed79-ba02-4dcc-b9a9-f409efda233a (at 192.168.202.53@tcp) reconnecting [ 3985.215935] Lustre: *** cfs_fail_loc=159, val=0*** [ 4001.876134] Lustre: lustre-MDT0001: Client b656ed79-ba02-4dcc-b9a9-f409efda233a (at 192.168.202.53@tcp) reconnecting [ 4004.320176] Lustre: *** cfs_fail_loc=159, val=0*** [ 4004.339955] Lustre: Skipped 3 previous similar messages [ 4020.764760] Lustre: lustre-MDT0001: Client b656ed79-ba02-4dcc-b9a9-f409efda233a (at 192.168.202.53@tcp) reconnecting [ 4036.385685] Lustre: 139155:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4038.941271] Lustre: *** cfs_fail_loc=159, val=0*** [ 4038.943497] Lustre: Skipped 4 previous similar messages [ 4054.539061] Lustre: lustre-MDT0000: Client a5b2887e-83d8-4779-bc1b-ce9b4d25836b (at 192.168.202.53@tcp) reconnecting [ 4056.790462] Lustre: *** cfs_fail_loc=159, val=0*** [ 4056.801086] Lustre: Skipped 15 previous similar messages [ 4072.483994] Lustre: lustre-MDT0000: Client a5b2887e-83d8-4779-bc1b-ce9b4d25836b (at 192.168.202.53@tcp) reconnecting [ 4074.203213] Lustre: 139452:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4094.955309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4094.960869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.993171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4095.001780] Lustre: Skipped 1 previous similar message [ 4099.408659] Lustre: server umount lustre-OST0000 complete [ 4105.189335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.193729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.200648] Lustre: Skipped 2 previous similar messages [ 4105.218431] Lustre: Skipped 1 previous similar message [ 4109.977214] Lustre: server umount lustre-MDT0000 complete [ 4110.322685] LustreError: 136087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.367993] LustreError: 136087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4114.208381] LustreError: 139277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387013 with bad export cookie 14513629530761088210 [ 4114.222771] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.227263] LustreError: 139277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4114.737307] Lustre: server umount lustre-MDT0001 complete [ 4126.190865] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4129.179948] Key type lgssc unregistered [ 4129.416646] LNet: 140534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4129.432774] LNetError: 140534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4129.461445] LNet: Removed LNI 192.168.202.153@tcp [ 4130.194080] Key type .llcrypt unregistered [ 4130.195704] Key type ._llcrypt unregistered [ 4152.785926] Key type ._llcrypt registered [ 4152.788399] Key type .llcrypt registered [ 4152.873971] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4154.247163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4154.299779] alg: No test for adler32 (adler32-zlib) [ 4155.259446] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4155.474079] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4157.128705] Key type lgssc registered [ 4158.186441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4166.307628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4166.328407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.925553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.782384] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4178.403630] LustreError: 141951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.429369] LustreError: 141951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4179.477483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.917741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4183.545675] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4189.933378] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4194.411713] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4200.428854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4200.783039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.930598] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4206.604130] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4213.068332] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4213.112779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4219.149563] Lustre: 144532:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4222.355555] Lustre: 144581:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4224.023074] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4225.090486] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4228.849542] Lustre: 144727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4229.900811] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4230.879375] Lustre: 144825:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4235.746438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.749520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.794976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.818842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.819022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.821787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4241.424359] Lustre: server umount lustre-OST0000 complete [ 4246.509671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4246.527579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.538922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.548799] Lustre: Skipped 1 previous similar message [ 4249.058477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.059042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.987691] Lustre: server umount lustre-MDT0000 complete [ 4254.178470] LustreError: 142678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.208251] LustreError: 142678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4255.993634] LustreError: 141933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387155 with bad export cookie 4518753432498544116 [ 4256.004963] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4256.011376] LustreError: 141933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4256.439786] Lustre: server umount lustre-MDT0001 complete [ 4268.914268] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4272.101920] Key type lgssc unregistered [ 4272.430831] LNet: 145886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4272.439381] LNetError: 145886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4272.452833] LNet: Removed LNI 192.168.202.153@tcp [ 4273.535963] Key type .llcrypt unregistered [ 4273.537877] Key type ._llcrypt unregistered [ 4301.601612] Key type ._llcrypt registered [ 4301.604778] Key type .llcrypt registered [ 4301.767841] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4302.539919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4302.617114] alg: No test for adler32 (adler32-zlib) [ 4303.720937] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4303.987689] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4305.624639] Key type lgssc registered [ 4307.316225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4315.584647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4315.616354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.117040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.158596] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4327.398135] LustreError: 147316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.415414] LustreError: 147316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4329.723102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.027826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4334.283545] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4342.601378] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4347.695410] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4355.803556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.740782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.867137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4364.825260] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4372.696895] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4375.179651] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4378.822161] Lustre: *** cfs_fail_loc=159, val=0*** [ 4395.044622] Lustre: lustre-MDT0000: Client 98ff561e-ca28-45dd-b060-c3a979a929dd (at 192.168.202.53@tcp) reconnecting [ 4413.409956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4413.430234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.441390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4416.488533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4416.497128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.521993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4418.777870] Lustre: server umount lustre-OST0000 complete [ 4426.722951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.737954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4426.742653] Lustre: Skipped 2 previous similar messages [ 4429.153388] Lustre: server umount lustre-MDT0000 complete [ 4431.842473] LustreError: 147316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.872335] LustreError: 147316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4433.164601] LustreError: 147296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387332 with bad export cookie 8536104731678221149 [ 4433.176913] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4433.181808] LustreError: 147296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4433.635832] Lustre: server umount lustre-MDT0001 complete [ 4445.764528] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4449.197929] Key type lgssc unregistered [ 4449.447347] LNet: 150981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4449.461996] LNetError: 150981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4449.509428] LNet: Removed LNI 192.168.202.153@tcp [ 4450.522803] Key type .llcrypt unregistered [ 4450.524458] Key type ._llcrypt unregistered [ 4473.778873] Key type ._llcrypt registered [ 4473.790325] Key type .llcrypt registered [ 4473.924850] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4475.068050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4475.097631] alg: No test for adler32 (adler32-zlib) [ 4476.184269] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4476.414980] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4478.056401] Key type lgssc registered [ 4479.198979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4487.376899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4487.399072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.288667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.932366] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4499.430147] LustreError: 152396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.444503] LustreError: 152396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4503.364555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.068831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4507.686829] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4515.303443] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4521.683881] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4528.449360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4528.817181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.850747] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4533.943479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4535.979080] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4543.668416] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4563.797575] Lustre: lustre-MDT0001: evicting 3314290b-5ec2-4ff8-bbbd-6164e63f5336 (at 192.168.202.53@tcp) by administrative request [ 4563.814272] Lustre: Skipped 1 previous similar message [ 4595.699509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.699895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.726086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.731909] Lustre: Skipped 1 previous similar message [ 4599.858406] Lustre: server umount lustre-OST0000 complete [ 4605.933767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.939283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4605.955692] Lustre: Skipped 2 previous similar messages [ 4605.975674] Lustre: Skipped 1 previous similar message [ 4611.043164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.051981] Lustre: Skipped 1 previous similar message [ 4616.165816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4616.177617] Lustre: Skipped 1 previous similar message [ 4618.208524] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4618.625741] Lustre: server umount lustre-MDT0000 complete [ 4621.283213] LustreError: 152396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.298973] LustreError: 152396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4623.790349] LustreError: 152377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387523 with bad export cookie 17747405872011820405 [ 4623.814121] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.328322] Lustre: server umount lustre-MDT0001 complete [ 4637.973619] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4641.346986] Key type lgssc unregistered [ 4641.706190] LNet: 156466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4641.716868] LNetError: 156466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4642.791598] LNet: Removed LNI 192.168.202.153@tcp [ 4643.915551] Key type .llcrypt unregistered [ 4643.918104] Key type ._llcrypt unregistered [ 4714.074487] Key type ._llcrypt registered [ 4714.076496] Key type .llcrypt registered [ 4714.161199] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4715.180593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4715.234732] alg: No test for adler32 (adler32-zlib) [ 4716.307970] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4716.530142] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4718.184154] Key type lgssc registered [ 4719.272838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4726.935966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4726.961612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.442682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.141669] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4738.027620] LustreError: 158956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.048554] LustreError: 158956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4741.524261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.838348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4744.991369] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4751.076638] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4756.326212] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4763.286787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4763.735633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.829185] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4771.129540] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4772.986432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4781.074929] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4791.777090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4791.781777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.791812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4793.827398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4793.830772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4793.834505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.281991] Lustre: server umount lustre-OST0000 complete [ 4803.553813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4803.560049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.581702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4803.588031] Lustre: Skipped 1 previous similar message [ 4807.096251] Lustre: server umount lustre-MDT0000 complete [ 4809.190872] LustreError: 158956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.214229] LustreError: 158956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4811.105255] LustreError: 158936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387710 with bad export cookie 767328519739319670 [ 4811.110763] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4811.133575] LustreError: 158936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4811.678080] Lustre: server umount lustre-MDT0001 complete [ 4824.080837] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4827.293684] Key type lgssc unregistered [ 4827.633747] LNet: 162496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4827.639865] LNetError: 162496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4828.712470] LNet: Removed LNI 192.168.202.153@tcp [ 4829.732982] Key type .llcrypt unregistered [ 4829.735363] Key type ._llcrypt unregistered [ 4855.762489] Key type ._llcrypt registered [ 4855.764193] Key type .llcrypt registered [ 4855.857955] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4857.371426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4857.477294] alg: No test for adler32 (adler32-zlib) [ 4858.709066] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 4859.161985] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4861.046942] Key type lgssc registered [ 4862.297669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4878.593925] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 04:36:15 (1787387775) [ 4896.476987] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4905.775205] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4942.521172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4942.561592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.910966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4943.976512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4944.102726] Lustre: lustre-MDT0000: new disk, initializing [ 4944.181828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.213367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4949.062389] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4960.202103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.289687] Lustre: 166759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4960.338440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4960.343225] Lustre: Skipped 1 previous similar message [ 4960.448247] Lustre: lustre-MDT0001: new disk, initializing [ 4960.552946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4960.646295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4960.674519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4965.409299] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4974.203623] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4979.714143] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4986.259719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.575434] Lustre: lustre-OST0000: new disk, initializing [ 4986.579940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4986.585856] Lustre: 168725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4986.679881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.875267] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4993.627218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4993.637349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4993.748686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5002.989691] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5014.507817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5014.515547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.522736] LustreError: Skipped 1 previous similar message [ 5014.523376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.564130] Lustre: Skipped 1 previous similar message [ 5019.620747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5019.640608] Lustre: Skipped 2 previous similar messages [ 5024.746617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5024.766053] Lustre: Skipped 1 previous similar message [ 5026.272211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5026.517257] Lustre: server umount lustre-OST0000 complete [ 5032.418495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5032.429724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.447734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5035.010929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.438245] Lustre: server umount lustre-MDT0000 complete [ 5041.442531] LustreError: 166751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387940 with bad export cookie 2825912463220908467 [ 5041.456324] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.457252] LustreError: 166751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5041.879263] Lustre: server umount lustre-MDT0001 complete [ 5052.831935] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5056.016029] Key type lgssc unregistered [ 5056.297414] LNet: 170635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5056.304165] LNetError: 170635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5057.324683] LNet: Removed LNI 192.168.202.153@tcp [ 5058.168219] Key type .llcrypt unregistered [ 5058.172147] Key type ._llcrypt unregistered [ 5084.314612] Key type ._llcrypt registered [ 5084.319450] Key type .llcrypt registered [ 5084.494901] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5086.064164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5086.084350] alg: No test for adler32 (adler32-zlib) [ 5087.260511] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 5087.604529] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5089.264229] Key type lgssc registered [ 5090.622275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5100.084682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5100.109237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.500080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.835488] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5111.778138] LustreError: 172049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.793442] LustreError: 172049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5116.773712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.907528] LustreError: 172050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.282287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.192787] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5130.266549] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5136.445271] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5146.019704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5146.661685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5147.711513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5155.439276] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5163.113810] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5170.852331] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5170.882947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.906080] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5170.915487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.928990] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5171.774762] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5171.793159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.803842] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5171.813949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.834090] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5172.812757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5172.838576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.856845] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5172.859792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.871629] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5173.698524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.692454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.707593] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5175.711452] Lustre: Skipped 2 previous similar messages [ 5175.723039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.740844] Lustre: Skipped 2 previous similar messages [ 5175.747498] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5175.750442] Lustre: Skipped 2 previous similar messages [ 5175.765148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.782360] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5175.792141] Lustre: Skipped 2 previous similar messages [ 5176.689825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.776120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.650411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.557715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.487634] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5180.493496] Lustre: Skipped 4 previous similar messages [ 5180.506757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.524548] Lustre: Skipped 4 previous similar messages [ 5180.528831] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5180.535025] Lustre: Skipped 4 previous similar messages [ 5180.542423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.555654] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5180.558851] Lustre: Skipped 4 previous similar messages [ 5181.555529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.337945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.416889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.251890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.169699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.151526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.594514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.381269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.365919] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5189.371990] Lustre: Skipped 8 previous similar messages [ 5189.385666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.393889] Lustre: Skipped 8 previous similar messages [ 5189.397967] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5189.402633] Lustre: Skipped 8 previous similar messages [ 5189.406602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.414087] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5189.416857] Lustre: Skipped 8 previous similar messages [ 5190.272336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.483951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.690761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.902444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.054987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.286062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.375390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.577857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.780244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.938217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.537866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.338327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.765329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5205.783149] Lustre: Skipped 12 previous similar messages [ 5205.824412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.833957] Lustre: Skipped 12 previous similar messages [ 5205.844835] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5205.852577] Lustre: Skipped 12 previous similar messages [ 5205.865084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.887480] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5205.899897] Lustre: Skipped 12 previous similar messages [ 5207.125393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.411829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.294310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.364853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.705433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.891125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.976320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.662366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.293848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.603965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.533612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.473329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.523359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.576898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.428554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.287087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.304397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.150528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.022708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.997377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.660990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.765472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.126201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.362747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.498197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.319994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.439189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.767739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.632491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.560704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.366155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.686480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5238.689635] Lustre: Skipped 31 previous similar messages [ 5238.699074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.706865] Lustre: Skipped 31 previous similar messages [ 5238.710424] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5238.714807] Lustre: Skipped 31 previous similar messages [ 5238.716752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.722247] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5238.729402] Lustre: Skipped 31 previous similar messages [ 5239.415776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.394363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.321821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.311234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.218818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.359855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.363704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.353321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.199787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.172327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.205537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.179361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.070086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.947645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.771988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.977210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.926973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.934769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.902189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.873249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.855641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.979865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.947079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.038592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.876033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.900776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.914767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.707073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.671870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.686323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.890040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.873111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.911956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.846572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.771158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.690207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.650995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.531961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.532821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.524580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.401652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.272153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.213675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.166100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.190892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.307927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.366117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.401722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.235918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.285205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.434964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.468702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.510971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.607374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.513252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.645550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.671730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.721648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.868358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.766704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.660275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.749464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.826919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.801061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.723880] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5302.728657] Lustre: Skipped 64 previous similar messages [ 5302.746808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5302.756462] Lustre: Skipped 64 previous similar messages [ 5302.763364] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5302.769980] Lustre: Skipped 64 previous similar messages [ 5302.773615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.789743] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5302.795834] Lustre: Skipped 64 previous similar messages [ 5303.663593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.577469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.362280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.347836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.240242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.292994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.116571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.157067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.960277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.096528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.976436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.919382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.657594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.544420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.454156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.526410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.325328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.299336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.220656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.191269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.988902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.966492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.827080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.633960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.507645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.537231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.445216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.466059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.741273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.738808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.724312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.601238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.727130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.509453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.764350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.723211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.730437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.703106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.627245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.653108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.793949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.890567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.172515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.096493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.733426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.289595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.266644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.974265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.060736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.168924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.354280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.839699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.628286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.517459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.961309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.723784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.003931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.296945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.264856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.636310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.804106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.931418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.029241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.028254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.007237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.853313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.801168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.646769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.645345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.674487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.649881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.363420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.285989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.250192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.388616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.187212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.105592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.037126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.861854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.868816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.983890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.257923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.339132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.324203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.462982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.517269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.572363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.757452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.765226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.664657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.694929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.724168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.680442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.717141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.615407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.467095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.508631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.469324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.464484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.396338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.123625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.019697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.732367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.692391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.575074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.556509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.565544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.446609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.257538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.415314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.539641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.379380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.345645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.347784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.364560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.468036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.539586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.453347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.388312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.259882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.272749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.182654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.216944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.245082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.281325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.261081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.243054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.425826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.322675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.350463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.326402] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5431.331369] Lustre: Skipped 130 previous similar messages [ 5431.345919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.366967] Lustre: Skipped 130 previous similar messages [ 5431.377373] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5431.391697] Lustre: Skipped 130 previous similar messages [ 5431.395754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.409951] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5431.420634] Lustre: Skipped 130 previous similar messages [ 5432.413650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.373987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.343627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.514981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.612302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.680689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.931958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.994805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.172325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.274862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.153309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.244523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.111106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.039767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.938254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.032334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.123437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.150662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.444214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.521843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.340211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.529334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.753753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.774840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.747905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.881939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.933893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.190900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.268979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.347929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.369056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.591101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.744426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.711694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.591197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.508682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.504570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.520347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.557021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.617379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.647614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.582916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.595896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.665258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.647298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.744831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.640171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.532322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.771318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.797597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.867938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.652859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.797835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.725203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.775851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.811584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.022740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.181985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.157964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.304620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.236116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.121709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.076661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.111632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.038051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.094648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.922492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.934159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.905281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.993470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.948083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.219472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.238883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.160209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.134481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.317173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.453932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.595052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.597974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.758787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.792026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.847347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.808308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.934311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.965612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.961681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.257167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.392412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.383089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.602838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.717706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.828777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.098788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.094543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.180180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.973394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.907842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.827844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.849309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.988680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.147042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.899618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.843477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.786595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.151351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.133257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.395387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.574672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.751895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.819453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.968462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.909824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.991441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.236296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.211120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.234154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.208845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.407428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.309866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.420412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.522294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.608688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.622977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.631783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.683137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.719114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.032077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.159485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.292361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.397276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.211790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.196135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.143139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.107479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.010744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.075686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.041456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.185487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.209276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.326594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.211836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.490217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.612514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.588918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.585263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.516349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.628941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.607433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.698679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.650675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.903305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.844526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.837829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.818200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.917616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.776729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.735966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.727407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.935592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.986954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.996712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.222556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.364495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.432752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.446941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.441245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.483923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.517366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.613499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.766945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.885604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.930624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.127150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.655501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.708360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.891541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.881951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.035931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.215864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.165993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.525951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.550684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.741058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.656381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.796492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.819239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.099785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.551016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.706823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.024163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.018312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.078712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.045670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.172404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.265817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.295178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.768101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.645916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.675121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.553177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.483072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.566805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.557730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.472714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.518494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.363696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.350442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.306756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.265852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.247514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.436136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.226439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.305254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.425287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.659172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.587678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.854996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.845282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.753516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.820416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.745669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.795944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.860644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.820508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.181287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.207048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.154662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.344179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.437843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.721866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.681437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.865841] LustreError: 172069:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5676.875874] LustreError: 172069:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5676.897485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.893534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.202569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.327700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.502655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.584565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.665531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.707392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.654105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.785981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.803357] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5687.817482] Lustre: Skipped 241 previous similar messages [ 5687.851193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5687.862354] Lustre: Skipped 241 previous similar messages [ 5687.875722] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5687.887303] Lustre: Skipped 241 previous similar messages [ 5687.900369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.921339] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5687.925726] Lustre: Skipped 241 previous similar messages [ 5689.091749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.258689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.197672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.399878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.329295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.471806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.432566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.660166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.940598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.441358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.645037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.962637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.003675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.952542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.018132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.991434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.166701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.319429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.374329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.468374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.674895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.688245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.648283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.761574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.882562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.786066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.799818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.767220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.833347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.786549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.787936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.827452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.961473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.775296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.645892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.635059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.576812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.096367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.881430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.826519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.112899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.197857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.178834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.615281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.842741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.355858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.721421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.350759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.253367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.005947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.071784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.576351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.514924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.597719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.576243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.427100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.356071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.454514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.304326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.287835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.577670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.148826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.266098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.679747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.242717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.578885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.905375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.402408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.842430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.835626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.883182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.928373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.139149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.875988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.552232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.143100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.031576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.280492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.042829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.868517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.544834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.331679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.234267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.240967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.239281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.012128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.568582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.507387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.456086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.363640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.389404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.381978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.462703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.484524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.858286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.498299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.622639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.375659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.280453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.354167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.279149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.607831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.085432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.354886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.401843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.412455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.428240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.241169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.309708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.347481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.656696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.846444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.193316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.681320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.008663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.320050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.106449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.460257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.390643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.619165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.184461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.997968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.998581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.137957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.958692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.121890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.060880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.703908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.351956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.598981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.878284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.858370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.330382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.432702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.685554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.862348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.922937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.892814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.979903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.400058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.551826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.451253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.594374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.338909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.006406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.954771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.075769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.131574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.665777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.989778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.356690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.423647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.700260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.664614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.885930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.104137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.158559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.144241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.377493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.618715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.946557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.279664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.447991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.625895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.412316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.694616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.850778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.142341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.284590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.046338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.971096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.814771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.595953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.554663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.179671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.150715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.022284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.138245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.349713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.419085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.213405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.419618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.698461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.832730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.084320] LustreError: 172069:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5887.099219] LustreError: 172069:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5887.112319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.576385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.917431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.234671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.496884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.056526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.705371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.915717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.084996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.219430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.150971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.245583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.852560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.018025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.349365] LustreError: 172069:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 5903.377406] LustreError: 172069:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 5903.381091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.537483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.490336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.647388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.573706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.503525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.620489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.733022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.734509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.014356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.287451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.382765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.405816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.612088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.731792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.909040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.056324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.081805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.146765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.006616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.178366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.305370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.474085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.372948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.253849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.438395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.285334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.390335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.520259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.716297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.861284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.921498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.259968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.409631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.684619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.776208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.773891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.727963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.679038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.676426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.568570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.672803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.666836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.663188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.559662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.824990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.692941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.510311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.434261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.301624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.343907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.330312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.387520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.449100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.411949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.492690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.598054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.603342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.447612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.587921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.518138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.489798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.300701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.354890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.166581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.269731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.421485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.417437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.270636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.273935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.183568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.142669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.296382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.003838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.099633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.173231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.957419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.955964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.088632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.334051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.313108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.656940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.713601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.912962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.992815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.948097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.832818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.573970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.871917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.708373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.082826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.137888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.236119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.204580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.178708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.471879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.778717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.741734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.880566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.973473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.005685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.301529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.557731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.270346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.744879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.970550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.126838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.366484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.601506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.105527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.007884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.081850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.814305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.551657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.409267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.279853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.270940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.535165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.719378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.714913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.705878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.843806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.959324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.036985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.220647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.621278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.592761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.695843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.586206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.541405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.490228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.880185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.136037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.365380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.550393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.295135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.410644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.895740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.961075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.065886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.218794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.508666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.599919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.741891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.167696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.412652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.906289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.197879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.227060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.387282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.308932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.389144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.183037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.138036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.784205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.526280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.172845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.143479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.999052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.792338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.774726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.024091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.032481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.112706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.359259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.513340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.568764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.497343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.359126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.378457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.123346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.221353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.039983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.311461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.389086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.385073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.592508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.525677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.507342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.860945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.923826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.894666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.918000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.110459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.207096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.029485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.221523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.355511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.264401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.732185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.971955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.005721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.190905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.279609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.210768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.018637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.968312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.039587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.096356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.266135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.275327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.444199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.783453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.106093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.169351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.123390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.208788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.491199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.674661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.710447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.892586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.960663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.178719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.187708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.410477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.755105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.892307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.140820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.181490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.508038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.685843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.714820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.816742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.016692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.980797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.208565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.214818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.431399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.343947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.514858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.624310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.666445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.898204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.353415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.549352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.736919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.571727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.875996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.946564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.959032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.965789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.980337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.941123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.956398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.019555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.149097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.428426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.598602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.883742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.226122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.206334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.460139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.328439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.521779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.048332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.131632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.291601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.296166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.218789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.264933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.387307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.480517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.640487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.727778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.757883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.675665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.641547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.812812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.849867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.159832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.512041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.459782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.521192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.637712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.923576] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6199.934321] Lustre: Skipped 472 previous similar messages [ 6199.965065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.984676] Lustre: Skipped 472 previous similar messages [ 6199.994873] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6200.002987] Lustre: Skipped 472 previous similar messages [ 6200.020185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.036855] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6200.045595] Lustre: Skipped 472 previous similar messages [ 6201.306426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.326340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.560804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.891763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.862808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.721323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.495491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.499835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.545065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.479784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.608354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.417458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.420659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.528562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.575042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.810142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.805296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.059931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.152593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.175529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.388153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.675717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.595585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.849965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6620.937530] LustreError: 173871:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6620.967441] LustreError: 172069:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7307.233556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7307.242049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.273854] Lustre: Skipped 24 previous similar messages [ 7307.286754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7307.787459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7309.545454] Lustre: server umount lustre-OST0000 complete [ 7317.988646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.995782] Lustre: Skipped 1 previous similar message [ 7320.907844] Lustre: server umount lustre-MDT0000 complete [ 7323.106105] LustreError: 172049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7323.136045] LustreError: 172049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7324.865860] LustreError: 172030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390224 with bad export cookie 18105930982949413030 [ 7324.869593] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7324.875918] LustreError: 172030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7325.463463] Lustre: server umount lustre-MDT0001 complete [ 7337.311197] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7340.211303] Key type lgssc unregistered [ 7340.515936] LNet: 227338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7340.538986] LNetError: 227338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7340.580084] LNet: Removed LNI 192.168.202.153@tcp [ 7341.574635] Key type .llcrypt unregistered [ 7341.577107] Key type ._llcrypt unregistered [ 7369.822582] Key type ._llcrypt registered [ 7369.825711] Key type .llcrypt registered [ 7369.928743] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7371.525795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7371.563314] alg: No test for adler32 (adler32-zlib) [ 7372.647233] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7372.963470] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7374.681900] Key type lgssc registered [ 7376.158499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7385.125783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7385.147785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.540085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7391.559906] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7395.816670] LustreError: 228755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.845318] LustreError: 228755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7400.379277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7400.837737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7406.488124] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7414.537397] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7420.017931] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7427.275164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7427.578937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.692928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7433.032499] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7440.877657] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7449.685682] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7451.720391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7451.729770] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7451.737944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7451.747690] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7451.747802] LustreError: 228774:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7453.694790] Lustre: lustre-OST0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 7455.203653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7455.212797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.229623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.254555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.257640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.295218] Lustre: Skipped 1 previous similar message [ 7459.941499] Lustre: server umount lustre-OST0000 complete [ 7471.776142] LustreError: 228774:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7495.406517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.747174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.768026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7497.773891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7502.426991] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7507.000788] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7507.016252] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7511.278109] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7513.069170] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7519.612420] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b0884e49000.ost_server_uuid 50 [ 7521.227567] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b0884e49000.ost_server_uuid in FULL state after 0 sec [ 7526.882257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7526.896074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.927908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7526.947154] Lustre: Skipped 1 previous similar message [ 7528.533490] Lustre: server umount lustre-OST0000 complete [ 7534.054751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7534.061660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.073029] Lustre: Skipped 1 previous similar message [ 7534.079273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7534.089685] Lustre: Skipped 2 previous similar messages [ 7542.754187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.768184] Lustre: Skipped 3 previous similar messages [ 7547.360222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7547.606407] Lustre: server umount lustre-MDT0000 complete [ 7551.976376] LustreError: 228749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.011249] LustreError: 228749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7552.142816] LustreError: 229490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390451 with bad export cookie 2941789286911966363 [ 7552.143274] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7552.159253] LustreError: 229490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7552.458992] Lustre: server umount lustre-MDT0001 complete [ 7565.082226] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7568.000864] Key type lgssc unregistered [ 7568.302929] LNet: 233961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.308520] LNetError: 233961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7569.385250] LNet: Removed LNI 192.168.202.153@tcp [ 7570.657501] Key type .llcrypt unregistered [ 7570.659937] Key type ._llcrypt unregistered [ 7608.888158] Key type ._llcrypt registered [ 7608.891838] Key type .llcrypt registered [ 7609.014424] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7610.328282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7610.389164] alg: No test for adler32 (adler32-zlib) [ 7611.610397] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 7611.973229] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7613.776208] Key type lgssc registered [ 7615.016165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7624.738446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7624.776390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.330979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.671730] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7636.463448] LustreError: 235690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.494143] LustreError: 235690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7641.203474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7641.570391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7641.760367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7647.102546] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7650.440781] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7657.332169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.974946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.946530] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7671.294226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7673.906353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.151567] Lustre: lustre-OST0001: new disk, initializing [ 7674.156848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7674.166064] Lustre: 237999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7674.277692] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.747365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7679.756774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7679.880653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7681.954851] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7693.490505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7700.976295] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7718.889863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7718.903051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.916844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7720.930772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7720.931257] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.962628] Lustre: Skipped 1 previous similar message [ 7723.173520] Lustre: server umount lustre-MDT0000 complete [ 7726.059833] LustreError: 235686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.089494] LustreError: 235686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7726.967674] LustreError: 235670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390626 with bad export cookie 14116239496209887276 [ 7726.969677] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7726.982274] LustreError: 235670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7727.311228] Lustre: server umount lustre-MDT0001 complete [ 7741.672202] Lustre: server umount lustre-OST0000 complete [ 7756.183571] Lustre: server umount lustre-OST0001 complete [ 7783.097289] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7795.628900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.486238] LustreError: 241107:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7796.603644] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7801.830289] LustreError: 241108:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.375168] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7805.923092] LustreError: 241107:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.490668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.867662] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.452957] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7821.235128] Lustre: 242250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7829.521684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.949705] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.114714] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7836.912379] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7837.166688] LustreError: 242603:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.190270] LustreError: 242603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7841.284217] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7845.109540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.491051] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7852.411469] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7861.489487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7865.393661] Lustre: 244118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7878.627570] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7878.648797] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.668593] Lustre: Skipped 1 previous similar message [ 7878.680436] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.686064] Lustre: Skipped 3 previous similar messages [ 7881.194988] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.226585] Lustre: Skipped 2 previous similar messages [ 7883.943378] Lustre: server umount mylustre-MDT0000 complete [ 7886.311455] LustreError: 241108:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.361532] LustreError: 241108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7888.173100] LustreError: 244321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390787 with bad export cookie 14116239496209890132 [ 7888.176663] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.179689] LustreError: 244321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7888.728887] Lustre: server umount mylustre-MDT0001 complete [ 7904.593623] Lustre: server umount mylustre-OST0000 complete [ 7907.433289] Lustre: 234955:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390790/real 1787390790] req@ffff981e8677fb80 x1874214792626560/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7907.518952] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.765568] Lustre: server umount mylustre-OST0001 complete [ 7936.844430] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7948.887910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.568288] LustreError: 246436:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.704593] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.713206] Lustre: Skipped 1 previous similar message [ 7953.413045] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7963.109599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7968.826191] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7972.163457] Lustre: 247579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7978.506875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.447432] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7988.102083] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 7994.385962] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7997.126534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.559282] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8002.643614] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8009.881660] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8013.901956] Lustre: 249452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8023.014451] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8023.055684] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8023.064519] Lustre: Skipped 6 previous similar messages [ 8028.005258] Lustre: server umount tfs-MDT0000 complete [ 8028.137109] LustreError: 247603:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.164366] LustreError: 247603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 8031.253799] LustreError: 246412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390930 with bad export cookie 14116239496209892316 [ 8031.272170] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8031.280464] LustreError: 246412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8031.808151] Lustre: server umount tfs-MDT0001 complete [ 8046.713700] Lustre: server umount tfs-OST0000 complete [ 8049.633030] Lustre: 234955:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787390932/real 1787390932] req@ffff981d82601f80 x1874214792677760/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787390948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8049.672168] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8049.697778] Lustre: Skipped 3 previous similar messages [ 8051.140150] Lustre: server umount tfs-OST0001 complete [ 8074.450745] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8084.397264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.040381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8085.043352] Lustre: Skipped 3 previous similar messages [ 8090.138713] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8100.035930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.323698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8106.662470] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8110.060094] Lustre: 252843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8117.688914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8125.248192] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8126.011099] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8130.362167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8134.617062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.302739] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8142.061863] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8150.047086] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8153.647226] Lustre: 254716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8161.469645] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 05:30:58 (1787391058) [ 8181.247560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.253842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8181.267634] Lustre: Skipped 2 previous similar messages [ 8181.286672] Lustre: Skipped 3 previous similar messages [ 8183.819830] Lustre: server umount lustre-MDT0000 complete [ 8186.353561] LustreError: 251703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.376698] LustreError: 251703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8187.663260] LustreError: 253221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391086 with bad export cookie 14116239496209894234 [ 8187.673690] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.680670] LustreError: 253221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8194.383321] Lustre: server umount lustre-MDT0001 complete [ 8205.054561] Lustre: server umount lustre-OST0000 complete [ 8215.904397] Lustre: server umount lustre-OST0001 complete [ 8223.883276] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8234.976614] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8278.014894] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8278.014894] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8278.014894] [ 8278.102120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8278.382816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8278.407905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8278.474603] Lustre: lustre-MDT0000: new disk, initializing [ 8278.550985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.556110] Lustre: Skipped 3 previous similar messages [ 8278.567484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8284.154739] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8296.037342] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8296.037342] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8296.037342] [ 8296.095133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8296.254801] Lustre: 257903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8296.313324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8296.317985] Lustre: Skipped 1 previous similar message [ 8296.438279] Lustre: lustre-MDT0001: new disk, initializing [ 8296.587300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8296.606832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8301.655406] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8314.176299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.578171] Lustre: lustre-OST0000: new disk, initializing [ 8314.583067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8314.591447] Lustre: 259489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8315.720974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8315.726460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8315.877340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8322.931820] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8336.427789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8336.616050] Lustre: lustre-OST0001: new disk, initializing [ 8336.621412] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8336.629809] Lustre: 260362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8338.171256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8338.193062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8338.371676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8344.877661] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8360.248988] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 05:34:17 (1787391257) [ 8378.507380] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:34:35 (1787391275) [ 8397.803571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8397.806787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8397.819332] LustreError: Skipped 1 previous similar message [ 8397.820258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.820263] Lustre: Skipped 4 previous similar messages [ 8397.920330] Lustre: Skipped 4 previous similar messages [ 8400.232452] Lustre: server umount lustre-OST0000 complete [ 8409.065147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8420.321028] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8420.634748] Lustre: server umount lustre-MDT0000 complete [ 8425.457694] LustreError: 257893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391324 with bad export cookie 14116239496209896012 [ 8425.467259] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8425.474793] LustreError: 257893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8432.015474] Lustre: server umount lustre-MDT0001 complete [ 8446.595662] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8459.163991] Lustre: server umount lustre-OST0001 complete [ 8467.880764] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8488.383972] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8529.819950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.076701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8530.101359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8530.206429] Lustre: lustre-MDT0000: new disk, initializing [ 8530.338622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8535.996465] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8547.788627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.896619] Lustre: 264934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8547.968202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8547.972832] Lustre: Skipped 1 previous similar message [ 8548.095682] Lustre: lustre-MDT0001: new disk, initializing [ 8548.177654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8548.187482] Lustre: Skipped 4 previous similar messages [ 8548.236954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8548.249855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8553.331307] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8564.639409] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8571.274537] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8579.230166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8579.616176] Lustre: 266901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8581.229855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8581.265363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8581.507365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8587.793326] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8598.448975] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8607.205440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.216504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8607.216982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8607.216986] Lustre: Skipped 14 previous similar messages [ 8607.218671] Lustre: Skipped 5 previous similar messages [ 8612.343213] LustreError: 266988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8612.371031] LustreError: 266988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 8612.728503] Lustre: server umount lustre-OST0000 complete [ 8623.946850] Lustre: server umount lustre-MDT0000 complete [ 8628.896140] LustreError: 264926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391528 with bad export cookie 14116239496209897475 [ 8628.903187] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.912797] LustreError: 264926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8629.452219] Lustre: server umount lustre-MDT0001 complete [ 8642.021546] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8645.325631] Key type lgssc unregistered [ 8645.611877] LNet: 268663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8645.618994] LNetError: 268663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8645.634970] LNet: Removed LNI 192.168.202.153@tcp [ 8646.720951] Key type .llcrypt unregistered [ 8646.727562] Key type ._llcrypt unregistered [ 8680.174849] Key type ._llcrypt registered [ 8680.176372] Key type .llcrypt registered [ 8680.334836] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8681.853392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8682.007098] alg: No test for adler32 (adler32-zlib) [ 8683.349606] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8683.862275] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8685.528186] Key type lgssc registered [ 8687.127238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8696.017751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8696.045092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8697.506419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8702.868986] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8707.042439] LustreError: 270130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8707.080471] LustreError: 270130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8712.174882] LustreError: 270131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.473700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.550307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.141049] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8733.061956] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8739.644251] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8748.072502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.606411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.200701] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8763.734960] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8765.237701] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8767.113623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8770.018916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8770.030863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.067020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8772.580959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8772.582358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.587609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8774.029949] Lustre: server umount lustre-OST0000 complete [ 8782.315271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8782.323917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.371141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8782.387963] Lustre: Skipped 1 previous similar message [ 8786.259406] Lustre: server umount lustre-MDT0000 complete [ 8787.938083] LustreError: 270130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.682359] LustreError: 270112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391689 with bad export cookie 113933006837272166 [ 8790.688626] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.702590] LustreError: 270112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8791.349098] Lustre: server umount lustre-MDT0001 complete [ 8804.905388] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8808.580986] Key type lgssc unregistered [ 8808.955586] LNet: 273728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8808.959565] LNetError: 273728:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8810.023928] LNet: Removed LNI 192.168.202.153@tcp [ 8811.008710] Key type .llcrypt unregistered [ 8811.011628] Key type ._llcrypt unregistered [ 8830.799429] Key type ._llcrypt registered [ 8830.802565] Key type .llcrypt registered [ 8831.009371] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8832.665411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8832.713360] alg: No test for adler32 (adler32-zlib) [ 8833.923337] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8834.235673] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8836.024184] Key type lgssc registered [ 8837.544409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8850.243197] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8850.263357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8865.825585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8870.944330] LustreError: 274919:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 8897.632390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.424315] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8918.710919] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8921.151428] Lustre: server umount lustre-OST0000 complete [ 8940.095540] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8942.850973] Key type lgssc unregistered [ 8943.153726] LNet: 276551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8943.169963] LNetError: 276551:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8943.191464] LNet: Removed LNI 192.168.202.153@tcp [ 8943.937716] Key type .llcrypt unregistered [ 8943.939676] Key type ._llcrypt unregistered [ 8984.382335] Key type ._llcrypt registered [ 8984.389651] Key type .llcrypt registered [ 8984.508757] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8985.688943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8985.759304] alg: No test for adler32 (adler32-zlib) [ 8986.994259] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 8987.244460] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8988.928466] Key type lgssc registered [ 8990.259375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9003.015683] loop0: detected capacity change from 0 to 204800000 [ 9003.219193] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9003.229293] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9003.731885] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9004.929116] loop0: detected capacity change from 0 to 204800000 [ 9005.133708] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9005.142794] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9005.516462] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9006.351606] loop0: detected capacity change from 0 to 204800000 [ 9006.558983] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9006.590592] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9006.764613] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9007.835201] loop0: detected capacity change from 0 to 204800000 [ 9008.018414] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9009.274276] loop0: detected capacity change from 0 to 204800000 [ 9009.299694] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9013.268138] loop0: detected capacity change from 0 to 204800000 [ 9013.305980] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9016.952631] loop0: detected capacity change from 0 to 204800000 [ 9017.024627] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9021.149315] loop0: detected capacity change from 0 to 204800000 [ 9021.185463] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9025.692723] loop0: detected capacity change from 0 to 204800000 [ 9025.769728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9025.836628] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9025.904466] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9025.915760] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9025.927240] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9025.934422] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9025.941806] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9025.956382] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9025.966257] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9025.974698] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9025.982228] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9025.991230] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9026.000564] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9026.009604] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9026.018634] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9026.035841] Lustre: 279966:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9031.891407] Lustre: server umount MGS complete [ 9032.854698] loop0: detected capacity change from 0 to 204800000 [ 9032.906934] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9033.692669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9033.758216] Lustre: lustre-MDT0000: Aborting client recovery [ 9034.790116] loop1: detected capacity change from 0 to 204800000 [ 9035.053400] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9035.176813] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9035.201978] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9035.234815] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9035.269604] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9035.286636] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9035.299634] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9035.315576] Lustre: 280397:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9035.364646] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9035.714031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9035.787740] Lustre: lustre-MDT0001: Aborting client recovery [ 9036.728145] loop2: detected capacity change from 0 to 204800000 [ 9036.875044] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9036.976669] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9037.004190] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9037.026376] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9037.037510] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9037.049468] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9037.070163] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9037.097598] Lustre: 280476:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9037.270498] Lustre: Found index 0 for lustre-OST0000, updating log [ 9037.314700] Lustre: 280494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9037.376403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9037.395401] Lustre: lustre-OST0000: Aborting recovery [ 9038.289326] loop3: detected capacity change from 0 to 204800000 [ 9038.558774] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9038.627584] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9038.635184] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9038.646504] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9038.653968] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9038.661831] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9038.669439] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9038.679211] Lustre: 280560:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9038.707657] Lustre: Found index 1 for lustre-OST0001, updating log [ 9038.729803] Lustre: 280566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9038.858303] Lustre: lustre-OST0001: Aborting recovery [ 9042.934676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9043.010559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9043.023933] LustreError: 280341:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9043.030598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9043.118127] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9044.978499] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9044.984798] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9048.735504] LustreError: 280308:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 12906us: rc = -2 [ 9049.267973] LustreError: 280309:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 479us: rc = -2 [ 9049.280707] LustreError: 280309:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 9065.458669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9065.461891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9065.479237] Lustre: Skipped 2 previous similar messages [ 9065.485371] Lustre: Skipped 3 previous similar messages [ 9068.393460] Lustre: server umount lustre-MDT0000 complete [ 9069.246903] LustreError: 280293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391968 with bad export cookie 16463241178144015028 [ 9069.264203] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9069.287016] LustreError: 280293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9070.560829] LustreError: 280305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9070.560953] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.562810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9070.585474] LustreError: 280305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9070.620541] Lustre: Skipped 2 previous similar messages [ 9074.594080] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9074.593844] LustreError: 280853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9074.605852] Lustre: Skipped 2 previous similar messages [ 9074.648982] LustreError: 280853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9075.818798] Lustre: server umount lustre-MDT0001 complete [ 9083.628746] Lustre: server umount lustre-OST0000 complete [ 9099.235593] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9099.496530] Lustre: server umount lustre-OST0001 complete [ 9110.016866] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 05:46:47 (1787392007) [ 9142.934519] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9153.505526] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9191.460240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.849971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9191.889619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9191.948953] Lustre: lustre-MDT0000: new disk, initializing [ 9192.026975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.032079] Lustre: Skipped 1 previous similar message [ 9192.066591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9197.575653] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9209.952608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.027227] Lustre: 284634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9210.060871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9210.063577] Lustre: Skipped 1 previous similar message [ 9210.105533] Lustre: lustre-MDT0001: new disk, initializing [ 9210.186691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9210.246774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9210.252379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9216.270301] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9227.007722] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9233.236667] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9240.531989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9240.805203] Lustre: lustre-OST0000: new disk, initializing [ 9240.808986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9240.814865] Lustre: 286604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9240.894604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.926542] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9250.429368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9250.436719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9250.572861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9259.350591] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9265.898268] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9277.166229] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9285.348236] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9337.832347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9337.844889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.854754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9337.855672] LustreError: Skipped 1 previous similar message [ 9337.866248] Lustre: Skipped 1 previous similar message [ 9341.238531] Lustre: server umount lustre-OST0000 complete [ 9348.076649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.087343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9348.105962] Lustre: Skipped 1 previous similar message [ 9348.118225] Lustre: Skipped 2 previous similar messages [ 9351.717693] Lustre: server umount lustre-MDT0000 complete [ 9353.193121] LustreError: 284647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.216231] LustreError: 284647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9355.452742] LustreError: 284628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392254 with bad export cookie 16463241178144017765 [ 9355.462803] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9355.902390] Lustre: server umount lustre-MDT0001 complete [ 9364.203675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.924143] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9374.321822] Lustre: server umount MGS complete [ 9385.579392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.229712] LustreError: 290470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.278590] LustreError: 290483:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9386.408339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.587596] LustreError: 290471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9392.653596] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9401.830373] LustreError: 290470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9401.848898] LustreError: 290470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9402.821074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.328313] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9415.696438] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9421.061702] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9428.168035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9428.591452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.595851] Lustre: Skipped 1 previous similar message [ 9436.707037] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9447.571458] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9478.115775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9478.119937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9478.129366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9479.137659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9484.261353] LustreError: 292293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.372737] Lustre: server umount lustre-OST0000 complete [ 9489.377688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.408819] Lustre: Skipped 2 previous similar messages [ 9494.383210] Lustre: server umount lustre-MDT0000 complete [ 9498.042726] LustreError: 290450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392397 with bad export cookie 16463241178144019606 [ 9498.050084] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9498.054525] LustreError: 290450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9498.555982] Lustre: server umount lustre-MDT0001 complete [ 9510.050701] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9513.253177] Key type lgssc unregistered [ 9513.801419] LNet: 294415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9513.823582] LNetError: 294415:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9513.839717] LNet: Removed LNI 192.168.202.153@tcp [ 9514.843872] Key type .llcrypt unregistered [ 9514.846417] Key type ._llcrypt unregistered [ 9555.633210] Key type ._llcrypt registered [ 9555.634827] Key type .llcrypt registered [ 9555.747215] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9571.670389] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9573.782030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9573.841330] alg: No test for adler32 (adler32-zlib) [ 9574.927150] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9575.329985] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 9577.240694] Key type lgssc registered [ 9579.338237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9612.805277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9612.859497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.182305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9614.245044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9614.356264] Lustre: lustre-MDT0000: new disk, initializing [ 9614.489413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.510580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9619.061852] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9630.078045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.231423] Lustre: 298375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9630.301373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9630.314779] Lustre: Skipped 1 previous similar message [ 9630.469711] Lustre: lustre-MDT0001: new disk, initializing [ 9630.620789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.671417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9630.690975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9635.893853] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9646.325050] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9652.474685] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9660.835182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.270162] Lustre: lustre-OST0000: new disk, initializing [ 9661.278296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9661.284188] Lustre: 300347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9661.480308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9668.022154] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9670.245898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9670.258236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9670.393597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9677.448858] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9685.056791] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9698.184515] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9709.670846] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9757.670075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.697482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9757.705361] Lustre: Skipped 1 previous similar message [ 9757.710509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9757.724660] Lustre: Skipped 1 previous similar message [ 9759.917807] Lustre: server umount lustre-OST0000 complete [ 9767.912842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.918150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9767.934850] Lustre: Skipped 1 previous similar message [ 9767.963454] Lustre: Skipped 1 previous similar message [ 9771.350683] Lustre: server umount lustre-MDT0000 complete [ 9776.102095] LustreError: 299172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392675 with bad export cookie 184343831152261749 [ 9776.108494] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9776.125501] LustreError: 299172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9776.565876] Lustre: server umount lustre-MDT0001 complete [ 9786.624913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.797831] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9797.375814] Lustre: server umount MGS complete [ 9809.847992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9810.376085] LustreError: 304402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.402387] LustreError: 304402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9810.412611] LustreError: 304415:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9810.497202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9815.121197] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9815.527835] LustreError: 304403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.645646] LustreError: 304402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.132295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.396545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9830.456670] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9839.699233] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9845.652057] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9854.023581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.407755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.771953] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9870.254020] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9896.940849] Lustre: server umount lustre-OST0000 complete [ 9902.056446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9902.064347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9902.075277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9905.635862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9905.634474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.756519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.775929] Lustre: Skipped 2 previous similar messages [ 9915.873043] Lustre: 296598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392799/real 1787392799] req@000000004a4b837c x1874216850963712/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787392815 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9915.891366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.384153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9916.916433] Lustre: server umount lustre-MDT0000 complete [ 9920.996336] LustreError: 304403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.025815] LustreError: 304403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9921.056671] Lustre: 296598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392804/real 1787392804] req@00000000692ec002 x1874216850965504/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787392820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9921.099830] Lustre: 296598:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9922.133304] LustreError: 304383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392821 with bad export cookie 184343831152263611 [ 9922.140350] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9922.152655] LustreError: 304383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9922.531872] Lustre: server umount lustre-MDT0001 complete [ 9937.220237] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9940.962250] Key type lgssc unregistered [ 9941.335638] LNet: 308352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9941.351684] LNetError: 308352:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9942.376529] LNet: Removed LNI 192.168.202.153@tcp [ 9943.415593] Key type .llcrypt unregistered [ 9943.417295] Key type ._llcrypt unregistered [ 9970.096727] Key type ._llcrypt registered [ 9970.099872] Key type .llcrypt registered [ 9970.263815] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9987.945547] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9989.624636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9989.755712] alg: No test for adler32 (adler32-zlib) [ 9991.019818] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [ 9991.422517] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 9993.200247] Key type lgssc registered [ 9994.692074] Lustre: Echo OBD driver; http://www.lustre.org/ [10021.422212] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10037.277044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10037.356079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.686771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10038.726644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10038.835117] Lustre: lustre-MDT0000: new disk, initializing [10038.947153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10038.966542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10043.946903] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10055.640460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.897488] Lustre: lustre-OST0000: new disk, initializing [10055.901173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10055.905202] Lustre: Skipped 1 previous similar message [10055.911723] Lustre: 312431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10055.987530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10061.963967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10061.975620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10062.016051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10062.567866] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10942.439873] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.450035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10947.554840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10947.965394] Lustre: server umount lustre-MDT0000 complete [10952.019576] LustreError: 311618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393851 with bad export cookie 598737235641590156 [10952.023743] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10952.031806] LustreError: 311618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10952.185990] Lustre: server umount lustre-OST0000 complete [10979.023642] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [10990.503462] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11030.055326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11030.364619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11030.401623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11030.499043] Lustre: lustre-MDT0000: new disk, initializing [11030.595758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11030.632549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11035.100770] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11046.445605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11046.645874] Lustre: 316772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11046.772932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11046.777143] Lustre: Skipped 1 previous similar message [11046.891252] Lustre: lustre-MDT0001: new disk, initializing [11046.997593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11047.057627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11047.072279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11051.952229] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11061.624361] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11068.223980] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11075.452662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11075.852168] Lustre: lustre-OST0000: new disk, initializing [11075.865653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11075.873646] Lustre: 318740:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11076.049641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11082.625160] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11085.890782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11085.895963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11086.032531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11091.359562] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11097.149634] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11097.395123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11103.314360] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11103.551125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11106.785293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.792068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.798427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.804986] Lustre: Skipped 1 previous similar message [11106.812930] Lustre: Skipped 1 previous similar message [11111.776731] Lustre: server umount lustre-OST0000 complete [11117.025469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.035700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11117.045510] Lustre: Skipped 1 previous similar message [11117.055233] Lustre: Skipped 1 previous similar message [11122.016472] Lustre: server umount lustre-MDT0000 complete [11127.217559] LustreError: 317566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394026 with bad export cookie 598737235644530975 [11127.219623] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11127.227964] LustreError: 317566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11127.766841] Lustre: server umount lustre-MDT0001 complete [11132.865136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11134.640641] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 06:20:32 (1787394032) [11143.454232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.142696] LustreError: 321052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.163457] LustreError: 321052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11144.294515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11147.733764] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11149.281991] LustreError: 321053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.404220] LustreError: 321052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11156.207874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.444832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11161.032435] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11168.359102] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11174.555536] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11181.109069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11181.498505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11188.485649] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11196.405460] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11204.095987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11204.247731] Lustre: lustre-OST0001: new disk, initializing [11204.250839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11204.255987] Lustre: 323888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11204.360638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11213.138949] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11213.509277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11213.530121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11213.664763] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11214.691827] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11214.716963] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11215.813838] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11215.835294] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11217.891230] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11217.931805] LustreError: 324257:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11222.009921] LustreError: 324268:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11222.038519] LustreError: 324268:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11226.882039] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11227.161913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11230.243484] LustreError: 324268:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11230.260317] LustreError: 324268:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11232.991125] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b0883971800.ost_server_uuid 50 [11234.631636] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b0883971800.ost_server_uuid in FULL state after 0 sec [11240.575235] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b0883971800.ost_server_uuid 50 [11242.056352] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b0883971800.ost_server_uuid in FULL state after 0 sec [11243.793253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11259.361181] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11259.361795] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.378137] LustreError: Skipped 2 previous similar messages [11259.384112] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11259.433438] Lustre: Skipped 1 previous similar message [11263.196959] Lustre: server umount lustre-OST0001 complete [11265.024114] LustreError: 322874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.039566] LustreError: 322874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11270.115421] LustreError: 323898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.151537] LustreError: 323898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11272.680178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11272.689757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.700466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11272.702790] Lustre: Skipped 2 previous similar messages [11277.713901] Lustre: server umount lustre-OST0000 complete [11284.453475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11284.460072] LustreError: Skipped 1 previous similar message [11284.463357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11284.471955] Lustre: Skipped 1 previous similar message [11288.623740] Lustre: server umount lustre-MDT0000 complete [11290.593651] LustreError: 321780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11290.611363] LustreError: 321780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11292.538776] LustreError: 323887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394191 with bad export cookie 598737235644531857 [11292.540293] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11292.545161] LustreError: 323887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11293.040914] Lustre: server umount lustre-MDT0001 complete [11305.868633] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11309.092853] Key type lgssc unregistered [11309.413963] LNet: 326578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11309.424944] LNetError: 326578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11309.467821] LNet: Removed LNI 192.168.202.153@tcp [11310.428986] Key type .llcrypt unregistered [11310.430688] Key type ._llcrypt unregistered [11340.795654] Key type ._llcrypt registered [11340.797738] Key type .llcrypt registered [11340.948782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11342.531853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11342.597947] alg: No test for adler32 (adler32-zlib) [11343.900848] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11344.173086] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11345.808218] Key type lgssc registered [11347.550785] Lustre: Echo OBD driver; http://www.lustre.org/ [11356.246125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11356.267143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.840264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11362.351224] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11368.427416] LustreError: 328001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.450650] LustreError: 328001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11371.508954] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11373.539751] LustreError: 328002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.683598] LustreError: 328001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.218243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.643556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11386.375942] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11397.201640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11397.685725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11402.795937] LustreError: 329681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.836455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11406.258762] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11409.901088] LustreError: 329682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.927879] LustreError: 329682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11414.573353] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11420.132316] LustreError: 329683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11420.156924] LustreError: 329683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11422.272793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11422.634801] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11427.852154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11429.307642] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11437.815919] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11447.858313] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11448.076247] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11550.689478] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11550.690920] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.701635] LustreError: Skipped 1 previous similar message [11550.704468] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11550.729105] Lustre: Skipped 1 previous similar message [11555.822872] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11555.830218] Lustre: Skipped 2 previous similar messages [11556.771532] Lustre: server umount lustre-OST0001 complete [11560.929213] LustreError: 329681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11563.503833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11563.516245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11563.538100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.050682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11566.053048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.070773] Lustre: Skipped 1 previous similar message [11568.701281] Lustre: server umount lustre-OST0000 complete [11575.265679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11575.276993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11575.306250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11579.007897] Lustre: server umount lustre-MDT0000 complete [11583.508313] LustreError: 327983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394482 with bad export cookie 10069903147258426871 [11583.509875] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11583.515383] LustreError: 327983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11584.063501] Lustre: server umount lustre-MDT0001 complete [11595.408814] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11598.284377] Key type lgssc unregistered [11598.589618] LNet: 332946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11598.593490] LNetError: 332946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11599.660622] LNet: Removed LNI 192.168.202.153@tcp [11600.843710] Key type .llcrypt unregistered [11600.846530] Key type ._llcrypt unregistered [11650.929901] Key type ._llcrypt registered [11650.932817] Key type .llcrypt registered [11651.051547] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11652.520201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11652.550133] alg: No test for adler32 (adler32-zlib) [11653.631301] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [11653.810223] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11655.457176] Key type lgssc registered [11656.651750] Lustre: Echo OBD driver; http://www.lustre.org/ [11667.405497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11667.426770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.277962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11675.254611] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11679.715941] LustreError: 335240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.745835] LustreError: 335240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11684.833927] LustreError: 335241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.069407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.739054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11692.131561] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11695.437508] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11701.701759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11702.224896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11710.518627] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11711.472281] LustreError: 336728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.541442] LustreError: 336727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.567826] LustreError: 336727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11714.575526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11719.662716] LustreError: 336728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.687649] LustreError: 336728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11720.570481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11720.881103] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11726.320983] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11727.875106] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11737.121575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11748.928912] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11763.682251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11763.686979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11763.700671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11767.277346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11767.290881] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.316735] Lustre: Skipped 2 previous similar messages [11769.263867] Lustre: server umount lustre-MDT0000 complete [11772.388733] LustreError: 335241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11772.401718] LustreError: 335241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11773.295937] LustreError: 335220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394672 with bad export cookie 13993425308495808691 [11773.305856] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11773.315664] LustreError: 335220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11773.856876] Lustre: server umount lustre-MDT0001 complete [11788.139387] Lustre: server umount lustre-OST0000 complete [11802.876398] Lustre: server umount lustre-OST0001 complete [11824.366687] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11836.204634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.510423] LustreError: 340509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.605403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.065636] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11850.748916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11851.329196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11857.606559] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11861.005753] Lustre: 341649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11867.633508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11867.974725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11873.929798] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11876.141554] LustreError: 342003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.162458] LustreError: 342003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11881.282570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11883.316847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11889.184607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11891.401689] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11899.552587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11903.618996] Lustre: 343520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11909.601557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11909.619667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11909.623410] Lustre: Skipped 3 previous similar messages [11909.627844] Lustre: Skipped 6 previous similar messages [11914.873313] Lustre: server umount lustre-MDT0000 complete [11918.790683] LustreError: 340490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394818 with bad export cookie 13993425308495810392 [11918.804496] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11919.338972] Lustre: server umount lustre-MDT0001 complete [11933.436782] Lustre: server umount lustre-OST0000 complete [11936.227331] Lustre: 334505:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394819/real 1787394819] req@ffff981ebfab5c00 x1874219030927360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11936.257354] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11936.797169] Lustre: server umount lustre-OST0001 complete [11940.338786] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11941.832389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11943.498518] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 06:34:01 (1787394841) [11965.858933] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11975.245237] loop0: detected capacity change from 0 to 1099511627776 [11975.739206] SGI XFS with ACLs, security attributes, quota, no debug enabled [11975.746980] *** VALIDATE xfs *** [11975.760082] XFS (loop0): Mounting V5 Filesystem [11975.795597] XFS (loop0): Ending clean mount [11977.341560] loop1: detected capacity change from 0 to 18691697672192 [12108.484294] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [12119.640923] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12157.016264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.274219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12157.301069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12157.363214] Lustre: lustre-MDT0000: new disk, initializing [12157.461602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12157.466070] Lustre: Skipped 1 previous similar message [12157.489248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12163.102456] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12174.678859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12174.848676] Lustre: 348676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12174.899254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12174.903341] Lustre: Skipped 1 previous similar message [12174.966697] Lustre: lustre-MDT0001: new disk, initializing [12175.188182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12175.210567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12180.444912] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12188.696442] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12193.932485] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12201.637516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12202.020556] Lustre: lustre-OST0000: new disk, initializing [12202.029680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12202.044857] Lustre: 350643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12203.528173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12203.541861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12203.694535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12208.728958] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12220.015782] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12225.300967] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12225.537160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12231.329219] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12231.590595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12234.220057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12234.225098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12234.231623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12234.266610] Lustre: Skipped 1 previous similar message [12239.339682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12239.350691] Lustre: Skipped 2 previous similar messages [12239.791600] Lustre: server umount lustre-OST0000 complete [12244.450912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12244.464212] Lustre: Skipped 1 previous similar message [12249.569191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12249.578637] Lustre: Skipped 3 previous similar messages [12250.773571] Lustre: server umount lustre-MDT0000 complete [12254.702941] LustreError: 349545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.733157] LustreError: 349545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [12255.227122] LustreError: 348668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395154 with bad export cookie 13993425308495811946 [12255.235718] LustreError: 348668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12255.244361] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12255.914518] Lustre: server umount lustre-MDT0001 complete [12264.339268] XFS (loop0): Unmounting Filesystem [12267.827137] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 06:39:25 (1787395165) [12276.733981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.673510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12277.676185] Lustre: Skipped 2 previous similar messages [12284.850101] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12294.240718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12301.267339] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12310.589142] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12317.368727] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12324.677381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12333.701137] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12343.213342] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12359.141130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.154427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12359.171757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12359.190290] Lustre: Skipped 1 previous similar message [12364.722405] Lustre: server umount lustre-OST0000 complete [12375.960219] Lustre: server umount lustre-MDT0000 complete [12380.918351] LustreError: 353184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395280 with bad export cookie 13993425308495812828 [12380.934238] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12380.941216] LustreError: 353184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12381.583912] Lustre: server umount lustre-MDT0001 complete [12394.878975] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [12397.602743] Key type lgssc unregistered [12397.906732] LNet: 356834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12397.911191] LNetError: 356834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12397.924351] LNet: Removed LNI 192.168.202.153@tcp [12398.521114] Key type .llcrypt unregistered [12398.522281] Key type ._llcrypt unregistered [12432.389538] Key type ._llcrypt registered [12432.391660] Key type .llcrypt registered [12432.516712] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12435.105530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12435.146191] alg: No test for adler32 (adler32-zlib) [12436.306282] Lustre: Lustre: Build Version: 2.17.54_114_g76b937b [12436.762937] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [12438.508345] Key type lgssc registered [12439.899435] Lustre: Echo OBD driver; http://www.lustre.org/ [12448.177164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12448.195128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.688844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12454.614030] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12458.982509] LustreError: 358407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.998409] LustreError: 358407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12463.658681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.079077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12469.050558] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12477.256805] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12483.358392] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12490.561731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12491.074284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12498.237822] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12507.026431] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12515.298207] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12515.302265] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.312491] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12519.918269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.946828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12519.964951] Lustre: Skipped 2 previous similar messages [12521.312784] Lustre: server umount lustre-MDT0001 complete [12525.033284] LustreError: 358402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.060312] LustreError: 358402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12530.151023] LustreError: 358403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.189544] LustreError: 358403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12533.823308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12533.948799] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12533.958341] Lustre: Found index 1 for lustre-MDT0001, updating log [12534.056546] Lustre: 358395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12534.549334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12538.856342] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12539.895077] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12541.942383] LustreError: 362004:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12541.944818] LustreError: 362004:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12541.951250] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12541.999073] LustreError: 362007:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -17 [12542.008063] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12542.008063] [12847.079148] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12847.092000] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12847.092390] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12847.104539] Lustre: Skipped 1 previous similar message [12850.899389] Lustre: server umount lustre-MDT0001 complete [12852.199347] LustreError: 358406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12852.214424] LustreError: 358406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12857.332812] LustreError: 359134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12857.362801] LustreError: 359134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12861.382931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12861.576574] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12861.581662] Lustre: Found index 1 for lustre-MDT0001, updating log [12861.638421] Lustre: 358394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12861.951901] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12867.093177] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12867.209983] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12871.167200] LustreError: 370580:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12871.176033] LustreError: 370580:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12871.184897] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12871.193065] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12871.273125] LustreError: 370583:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12871.279269] LustreError: 370583:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12871.285350] LustreError: 370583:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -17 [12871.299478] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.153@tcp [12871.299478] [13175.316583] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [13175.321609] Lustre: Skipped 1 previous similar message [13179.386821] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13179.395655] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.419847] Lustre: Skipped 1 previous similar message [13179.449209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13179.458942] Lustre: Skipped 1 previous similar message [13180.864238] Lustre: server umount lustre-MDT0001 complete [13184.502037] LustreError: 358401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13184.545086] LustreError: 358401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13193.645412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13193.737862] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13193.743491] Lustre: Found index 1 for lustre-MDT0001, updating log [13193.808317] Lustre: 358396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13194.307641] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13199.377586] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13199.892546] LustreError: 378734:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13199.897385] LustreError: 378734:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13199.903157] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13199.906259] LustreError: 358398:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13200.002146] LustreError: 378743:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13200.023854] LustreError: 378743:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -114 [13200.036831] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13200.036831] [13200.146192] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13527.013394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13527.018164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13527.021600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13527.021610] Lustre: Skipped 1 previous similar message [13527.055815] Lustre: Skipped 3 previous similar messages [13528.500672] Lustre: server umount lustre-MDT0000 complete [13532.132785] LustreError: 360182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13532.156955] LustreError: 360182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [13532.800511] LustreError: 358387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396432 with bad export cookie 9336627714627276999 [13532.819509] LustreError: 358387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13532.822535] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13537.258217] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13539.320970] Lustre: server umount lustre-MDT0001 complete [13544.236970] Lustre: server umount lustre-OST0000 complete [13554.656283] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [13571.962789] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [13615.537297] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 07:01:52 (1787396512) [13623.815152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13624.265826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13624.312393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13624.369832] Lustre: lustre-MDT0000: new disk, initializing [13624.452314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13624.478372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13630.470807] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13643.078515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13643.239411] Lustre: 389327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13643.319216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13643.323174] Lustre: Skipped 1 previous similar message [13643.387301] Lustre: lustre-MDT0001: new disk, initializing [13643.531676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13643.606985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13643.626054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13650.267116] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13660.645785] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13668.026507] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13676.962910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13677.243000] Lustre: lustre-OST0000: new disk, initializing [13677.248494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13677.258042] Lustre: 391297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13677.412727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13684.842339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13684.849049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13685.026268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13686.261769] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [13697.886368] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13710.311436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13710.322830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13710.346518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13710.357874] Lustre: Skipped 3 previous similar messages [13713.329159] Lustre: server umount lustre-MDT0000 complete [13715.943972] LustreError: 389337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13715.962183] LustreError: 389337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13718.416045] LustreError: 389317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396617 with bad export cookie 9336627714627278952 [13718.436732] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13718.441964] LustreError: 389317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13718.867980] Lustre: server umount lustre-MDT0001 complete [13734.677328] Lustre: server umount lustre-OST0000 complete [13768.809897] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [13783.290910] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [13822.993685] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13566 sec ========= 07:05:19 (1787396719) [13825.147463] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:05:22 (1787396722) === [13829.502577] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:05:26 (1787396726) === [13878.163901] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [13881.707141] Key type lgssc unregistered [13882.049470] LNet: 397666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13882.058357] LNetError: 397666:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13883.111654] LNet: Removed LNI 192.168.202.153@tcp [13884.323348] Key type .llcrypt unregistered [13884.326200] Key type ._llcrypt unregistered