[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484116113 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003201] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008362] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.013043] Yama: becoming mindful. [ 0.014038] SELinux: Initializing. [ 0.015058] *** VALIDATE selinux *** [ 0.023798] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028365] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] *** VALIDATE tmpfs *** [ 0.032408] *** VALIDATE proc *** [ 0.033255] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037105] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044141] debug: unmapping init [mem 0xffffffffab859000-0xffffffffab860fff] [ 0.046788] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055282] rcu: Hierarchical SRCU implementation. [ 0.057369] smp: Bringing up secondary CPUs ... [ 0.058623] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.062397] smp: Brought up 1 node, 4 CPUs [ 0.064012] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144139] node 0 deferred pages initialised in 76ms [ 0.147120] devtmpfs: initialized [ 0.148244] x86/mm: Memory block size: 128MB [ 0.150835] gcov: version magic: 0x41383552 [ 0.151581] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157387] pinctrl core: initialized pinctrl subsystem [ 0.160162] [ 0.160717] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ** ** [ 0.167011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170014] ** ** [ 0.172010] ** This means that this kernel is built to expose internal ** [ 0.175011] ** IOMMU data structures, which may compromise security on ** [ 0.177011] ** your system. ** [ 0.179011] ** ** [ 0.182013] ** If you see this message and you are not debugging the ** [ 0.184011] ** kernel, report this immediately to your vendor! ** [ 0.187011] ** ** [ 0.189071] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191008] ************************************************************* [ 0.193690] NET: Registered protocol family 16 [ 0.195399] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204098] cpuidle: using governor menu [ 0.205664] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208645] PCI: Using configuration type 1 for base access [ 0.210116] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227076] cryptd: max_cpu_qlen set to 1000 [ 0.229226] ACPI: Added _OSI(Module Device) [ 0.230023] ACPI: Added _OSI(Processor Device) [ 0.231028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237996] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244339] ACPI: Interpreter enabled [ 0.246060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247011] ACPI: Using IOAPIC for interrupt routing [ 0.249113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252374] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263646] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277010] acpiphp: Slot [2] registered [ 0.278126] acpiphp: Slot [5] registered [ 0.280109] acpiphp: Slot [6] registered [ 0.282116] acpiphp: Slot [7] registered [ 0.283134] acpiphp: Slot [8] registered [ 0.285125] acpiphp: Slot [9] registered [ 0.287110] acpiphp: Slot [10] registered [ 0.288108] acpiphp: Slot [3] registered [ 0.290120] acpiphp: Slot [4] registered [ 0.291118] acpiphp: Slot [11] registered [ 0.293135] acpiphp: Slot [12] registered [ 0.295174] acpiphp: Slot [13] registered [ 0.296094] acpiphp: Slot [14] registered [ 0.297095] acpiphp: Slot [15] registered [ 0.299106] acpiphp: Slot [16] registered [ 0.301119] acpiphp: Slot [17] registered [ 0.302106] acpiphp: Slot [18] registered [ 0.304111] acpiphp: Slot [19] registered [ 0.305099] acpiphp: Slot [20] registered [ 0.307094] acpiphp: Slot [21] registered [ 0.308167] acpiphp: Slot [22] registered [ 0.310123] acpiphp: Slot [23] registered [ 0.312106] acpiphp: Slot [24] registered [ 0.313147] acpiphp: Slot [25] registered [ 0.315096] acpiphp: Slot [26] registered [ 0.316083] acpiphp: Slot [27] registered [ 0.318083] acpiphp: Slot [28] registered [ 0.319145] acpiphp: Slot [29] registered [ 0.320087] acpiphp: Slot [30] registered [ 0.322086] acpiphp: Slot [31] registered [ 0.323093] PCI host bridge to bus 0000:00 [ 0.324016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342143] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346638] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359999] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.365016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.373018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375516] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378067] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384804] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.390019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416857] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.469116] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503034] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.513916] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.523013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.533016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.579536] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.586019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626280] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.633021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.639021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.659027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.672135] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.679020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.689021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.727063] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.729456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.734560] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.737480] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.740309] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.744202] iommu: Default domain type: Passthrough [ 0.746592] SCSI subsystem initialized [ 0.748215] ACPI: bus type USB registered [ 0.750137] usbcore: registered new interface driver usbfs [ 0.753131] usbcore: registered new interface driver hub [ 0.755114] usbcore: registered new device driver usb [ 0.757254] pps_core: LinuxPPS API ver. 1 registered [ 0.759016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.762088] PTP clock support registered [ 0.765120] EDAC MC: Ver: 3.0.0 [ 0.767136] PCI: Using ACPI for IRQ routing [ 0.768928] NetLabel: Initializing [ 0.770018] NetLabel: domain hash size = 128 [ 0.772016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.774102] NetLabel: unlabeled traffic allowed by default [ 0.776201] vgaarb: loaded [ 0.778212] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.780016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.785423] clocksource: Switched to clocksource kvm-clock [ 0.896713] VFS: Disk quotas dquot_6.6.0 [ 0.898587] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.901379] *** VALIDATE ramfs *** [ 0.902764] *** VALIDATE hugetlbfs *** [ 0.904551] pnp: PnP ACPI init [ 0.907838] pnp: PnP ACPI: found 6 devices [ 0.925412] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.928927] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.931396] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.933862] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.936970] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.939989] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.943140] NET: Registered protocol family 2 [ 0.945812] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.950933] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.954921] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.960626] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964336] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968519] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.971979] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975133] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978659] NET: Registered protocol family 1 [ 0.981409] RPC: Registered named UNIX socket transport module. [ 0.983614] RPC: Registered udp transport module. [ 0.985138] RPC: Registered tcp transport module. [ 0.986838] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.989152] NET: Registered protocol family 44 [ 0.991063] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.993724] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.995779] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.998032] PCI: CLS 0 bytes, default 64 [ 0.999768] Unpacking initramfs... [ 2.440772] debug: unmapping init [mem 0xffff8e90bcc54000-0xffff8e90bffbffff] [ 2.447701] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.450393] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.454959] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.961937] Initialise system trusted keyrings [ 2.964250] Key type blacklist registered [ 2.966687] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.976477] zbud: loaded [ 2.979626] *** VALIDATE nfs *** [ 2.981042] *** VALIDATE nfs4 *** [ 2.982633] pstore: using deflate compression [ 2.985822] Platform Keyring initialized [ 3.086621] NET: Registered protocol family 38 [ 3.088380] Key type asymmetric registered [ 3.089957] Asymmetric key parser 'x509' registered [ 3.091867] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.095158] io scheduler mq-deadline registered [ 3.096747] io scheduler kyber registered [ 3.098367] io scheduler bfq registered [ 3.100328] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.103521] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.106686] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.109721] ACPI: Power Button [PWRF] [ 3.114575] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.121937] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.135982] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.142847] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.165661] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.194063] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.224440] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.229753] Non-volatile memory driver v1.3 [ 3.231690] Linux agpgart interface v0.103 [ 3.262650] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.265156] vda: detected capacity change from 0 to 76689408 [ 3.279782] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.282988] vdb: detected capacity change from 0 to 1073741824 [ 3.297812] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.301016] vdc: detected capacity change from 0 to 2621440000 [ 3.317194] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.319893] vdd: detected capacity change from 0 to 2621440000 [ 3.335662] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.338500] vde: detected capacity change from 0 to 4294967296 [ 3.353694] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.356558] vdf: detected capacity change from 0 to 4294967296 [ 3.364838] libphy: Fixed MDIO Bus: probed [ 3.369897] usbcore: registered new interface driver usbserial_generic [ 3.372906] usbserial: USB Serial support registered for generic [ 3.375481] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.379821] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.381522] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.383921] mousedev: PS/2 mouse device common for all mice [ 3.387506] rtc_cmos 00:05: RTC can wake from S4 [ 3.389841] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.391854] rtc_cmos 00:05: registered as rtc0 [ 3.395385] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.398168] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.399124] intel_pstate: CPU model not supported [ 3.403865] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.409592] hid: raw HID events driver (C) Jiri Kosina [ 3.411537] usbcore: registered new interface driver usbhid [ 3.413305] usbhid: USB HID core driver [ 3.414870] drop_monitor: Initializing network drop monitor service [ 3.416825] Initializing XFRM netlink socket [ 3.418675] NET: Registered protocol family 10 [ 3.421925] Segment Routing with IPv6 [ 3.423328] NET: Registered protocol family 17 [ 3.425522] mpls_gso: MPLS GSO support [ 3.432074] RAS: Correctable Errors collector initialized. [ 3.434799] AVX version of gcm_enc/dec engaged. [ 3.436337] AES CTR mode by8 optimization enabled [ 3.516545] sched_clock: Marking stable (3516521361, 0)->(4424064322, -907542961) [ 3.520086] registered taskstats version 1 [ 3.521948] Loading compiled-in X.509 certificates [ 3.524136] zswap: loaded using pool lzo/zbud [ 3.548551] Key type big_key registered [ 3.563059] Key type encrypted registered [ 3.564545] ima: No TPM chip found, activating TPM-bypass! [ 3.566265] ima: Allocated hash algorithm: sha1 [ 3.568169] ima: No architecture policies found [ 3.569689] evm: Initialising EVM extended attributes: [ 3.571363] evm: security.selinux [ 3.572439] evm: security.ima [ 3.573581] evm: security.capability [ 3.574639] evm: HMAC attrs: 0x1 [ 3.577433] rtc_cmos 00:05: setting system clock to 2026-09-06 03:32:51 UTC (1788665571) [ 3.584886] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 3.588314] debug: unmapping init [mem 0xffffffffab582000-0xffffffffab858fff] [ 3.595185] Write protecting the kernel read-only data: 28672k [ 3.599301] debug: unmapping init [mem 0xffffffffa9c03000-0xffffffffa9dfffff] [ 3.602667] debug: unmapping init [mem 0xffffffffaa514000-0xffffffffaa5fffff] [ 3.635212] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.642347] systemd[1]: Detected virtualization kvm. [ 3.644787] systemd[1]: Detected architecture x86-64. [ 3.646861] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.671077] systemd[1]: No hostname configured. [ 3.672924] systemd[1]: Set hostname to . [ 3.675212] random: systemd: uninitialized urandom read (16 bytes read) [ 3.677751] systemd[1]: Initializing machine ID from random generator. [ 3.729693] random: ln: uninitialized urandom read (6 bytes read) [ 3.830438] random: systemd: uninitialized urandom read (16 bytes read) [ 3.833868] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.838128] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.842603] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.486681] device-mapper: uevent: version 1.0.3 [ 4.488770] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [[ 5.144023] random: fast init done  OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.194544] virtio_net virtio0 ens2: renamed from eth0 [ 5.335873] scsi host0: ata_piix [ 5.368479] scsi host1: ata_piix [ 5.369945] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.372316] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.727536] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.079527] random: crng init done [ 10.080744] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.414834] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.494816] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.764810] SELinux: Disabled at runtime. [ 11.823579] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.832111] systemd[1]: Detected virtualization kvm. [ 11.834212] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.324920] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.328049] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.332492] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.337295] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.341084] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.347895] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.352460] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ 12.520090] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.810292] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.058206] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.133843] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.196819] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.223190] EDAC sbridge: Ver: 1.1.2 [ 14.953237] Key type dns_resolver registered [ 15.254072] NFS: Registering the id_resolver key type [ 15.256270] Key type id_resolver registered [ 15.258043] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg633-server login: [ 39.053067] hrtimer: interrupt took 2076585 ns [ 62.980560] libcfs: loading out-of-tree module taints kernel. [ 63.035400] Key type ._llcrypt registered [ 63.039508] Key type .llcrypt registered [ 63.124417] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 81.547798] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 83.189515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.246307] alg: No test for adler32 (adler32-zlib) [ 84.851350] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 85.741108] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 87.432573] Key type lgssc registered [ 88.982726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.588229] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 141.115720] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 152.297935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.328443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.509945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.543982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.610775] Lustre: lustre-MDT0000: new disk, initializing [ 153.697094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.715281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.722835] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.751431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.938533] Lustre: 6532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.061453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.064924] Lustre: Skipped 1 previous similar message [ 175.134969] Lustre: lustre-MDT0001: new disk, initializing [ 175.228421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.266394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.279580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.325088] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.518437] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.578708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.886452] Lustre: lustre-OST0000: new disk, initializing [ 193.902546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.921180] Lustre: 8470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.004190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.679934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.689674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.775837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 201.638440] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.931159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.150143] Lustre: lustre-OST0001: new disk, initializing [ 218.157677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 218.167692] Lustre: 9545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.260763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.762954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.773894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.869424] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.443922] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.350261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.553901] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.852687] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.957378] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing yml_node [ 263.493226] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 266.397641] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 268.543649] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 270.016047] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 23:37:16 EDT 2026 [ 286.021730] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 287.548911] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.635682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.643337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.658939] Lustre: Skipped 1 previous similar message [ 310.534767] Lustre: server umount lustre-MDT0000 complete [ 314.852714] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665882 with bad export cookie 11601853534770513179 [ 314.854840] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.878653] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 315.874263] LustreError: 9060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.882100] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.897083] LustreError: 9060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 315.900803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.916836] Lustre: Skipped 3 previous similar messages [ 315.926673] Lustre: Skipped 3 previous similar messages [ 319.904966] LustreError: 6540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.909791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.933436] LustreError: 6540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 319.943583] Lustre: Skipped 2 previous similar messages [ 321.267090] Lustre: server umount lustre-MDT0001 complete [ 325.189138] Lustre: server umount lustre-OST0000 complete [ 328.699485] Lustre: server umount lustre-OST0001 complete [ 334.487693] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 343.136495] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 374.217122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.472193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.502960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.598782] Lustre: lustre-MDT0000: new disk, initializing [ 374.728953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.759929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 379.155637] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 389.689727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.820830] Lustre: 15094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 389.905752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 389.912901] Lustre: Skipped 1 previous similar message [ 390.042279] Lustre: lustre-MDT0001: new disk, initializing [ 390.168580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 390.205396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 390.230031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.216487] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 404.969952] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.699375] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.191435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.503888] Lustre: lustre-OST0000: new disk, initializing [ 418.508545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 418.521639] Lustre: 17063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.669950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.079370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 420.097159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.304925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.811368] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 434.666975] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 439.299536] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 439.514982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 444.423477] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 444.665030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.042199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.064466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.072292] Lustre: Skipped 1 previous similar message [ 452.851738] Lustre: server umount lustre-OST0000 complete [ 456.165920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.167063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.177700] Lustre: Skipped 1 previous similar message [ 456.192431] Lustre: Skipped 2 previous similar messages [ 462.484994] Lustre: server umount lustre-MDT0000 complete [ 465.996775] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666033 with bad export cookie 11601853534770514712 [ 465.998622] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 466.003178] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 466.351544] Lustre: server umount lustre-MDT0001 complete [ 471.057299] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 23:40:37 (1788666037) [ 477.032548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.413541] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.492281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.401312] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 482.790075] LustreError: 19327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.916717] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.858036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.845635] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 499.549934] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.209560] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.514873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.788538] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 527.553925] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 531.365770] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 537.574032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.575811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.585922] Lustre: Skipped 1 previous similar message [ 537.613651] Lustre: Skipped 4 previous similar messages [ 543.662461] Lustre: server umount lustre-MDT0000 complete [ 547.775604] LustreError: 20062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666115 with bad export cookie 11601853534770515587 [ 547.778648] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.806472] LustreError: 20062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 547.839945] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.840912] LustreError: 19321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.848695] Lustre: Skipped 1 previous similar message [ 548.164678] Lustre: server umount lustre-MDT0001 complete [ 557.917780] Lustre: server umount lustre-OST0000 complete [ 578.624098] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 588.567124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.914272] LustreError: 23706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.000785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.006090] Lustre: Skipped 2 previous similar messages [ 593.159259] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 602.953037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.113655] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 611.451402] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 618.819068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 627.399477] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 637.060227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.293701] Lustre: lustre-OST0001: new disk, initializing [ 637.299852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 637.308531] Lustre: 26022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 638.854178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 638.867979] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 639.015670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 644.289670] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 654.499457] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 657.937809] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 669.666657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.671949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 669.690796] Lustre: Skipped 2 previous similar messages [ 669.711424] Lustre: Skipped 7 previous similar messages [ 675.792576] Lustre: server umount lustre-MDT0000 complete [ 679.319731] LustreError: 23688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666247 with bad export cookie 11601853534770516441 [ 679.320590] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.330909] LustreError: 23688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 679.681738] Lustre: server umount lustre-MDT0001 complete [ 693.962355] Lustre: server umount lustre-OST0000 complete [ 695.265272] Lustre: 3665:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666247/real 1788666247] req@ffff8e9004c65c00 x1875551875995136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666263 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 698.062408] Lustre: server umount lustre-OST0001 complete [ 716.633669] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 727.694976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.222284] LustreError: 29043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 728.256093] LustreError: 29043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 728.335847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.349207] Lustre: Skipped 3 previous similar messages [ 733.081175] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 742.329407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.360355] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 750.359548] Lustre: 30193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 757.762102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 765.878203] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 775.593759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 783.682158] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 792.268282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 797.236900] Lustre: 32070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 806.893210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.895245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 806.943283] Lustre: Skipped 4 previous similar messages [ 806.947571] Lustre: Skipped 5 previous similar messages [ 811.021947] Lustre: server umount lustre-MDT0000 complete [ 812.009459] LustreError: 29040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 812.030163] LustreError: 29040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 815.159162] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666383 with bad export cookie 11601853534770517736 [ 815.165593] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 815.168931] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 815.668593] Lustre: server umount lustre-MDT0001 complete [ 830.375322] Lustre: server umount lustre-OST0000 complete [ 833.312201] Lustre: 3666:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666385/real 1788666385] req@ffff8e9002219500 x1875551876047616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 834.525674] Lustre: server umount lustre-OST0001 complete [ 844.261729] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 23:46:49 (1788666409) [ 871.632580] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 881.418723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.147801] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 896.026584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.956482] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 904.174763] Lustre: 36020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 912.070284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 919.918865] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 928.103198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 935.586239] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 944.958806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 949.445972] Lustre: 37899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 957.926835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 957.942748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.962762] Lustre: Skipped 2 previous similar messages [ 957.970772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 957.977314] Lustre: Skipped 2 previous similar messages [ 962.235477] Lustre: server umount lustre-MDT0000 complete [ 964.584249] LustreError: 36127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.635964] LustreError: 36127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 967.076946] LustreError: 38051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666534 with bad export cookie 11601853534770518842 [ 967.083991] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 967.103142] LustreError: 38051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 967.424680] Lustre: server umount lustre-MDT0001 complete [ 981.442546] Lustre: server umount lustre-OST0000 complete [ 995.542470] Lustre: server umount lustre-OST0001 complete [ 1003.633657] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 23:49:29 (1788666569) [ 1019.626585] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1030.288684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.783727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.801412] Lustre: Skipped 7 previous similar messages [ 1036.301021] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1044.602422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.830365] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1051.803607] Lustre: 41196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1058.311344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1064.689173] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1071.755929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.580276] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1083.078981] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1085.874190] Lustre: 43070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1086.614654] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1086.618788] Lustre: Skipped 1 previous similar message [ 1088.903422] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 1088.906613] Lustre: Skipped 4 previous similar messages [ 1091.042829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1091.049369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.058993] Lustre: Skipped 3 previous similar messages [ 1095.164472] Lustre: server umount lustre-MDT0000 complete [ 1098.877289] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666666 with bad export cookie 11601853534770519941 [ 1098.880233] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.884308] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1105.310768] Lustre: server umount lustre-MDT0001 complete [ 1114.750374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.297453] LustreError: 3664:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e91367ac000 x1875551876147200/t0(0) o250->MGC192.168.206.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1127.506029] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1133.997789] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1134.450203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.725815] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1140.214993] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1140.223871] Lustre: Skipped 1 previous similar message [ 1145.948894] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1147.742171] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1153.117853] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1154.659886] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1160.167216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1166.189923] Lustre: server umount lustre-MDT0000 complete [ 1169.798800] LustreError: 43073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666737 with bad export cookie 11601853534770521075 [ 1169.800799] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.811300] LustreError: 43073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1170.449947] Lustre: server umount lustre-MDT0001 complete [ 1184.759560] Lustre: server umount lustre-OST0000 complete [ 1186.785581] Lustre: 3665:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666738/real 1788666738] req@ffff8e90020fed80 x1875551876179456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788666754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1188.027555] Lustre: server umount lustre-OST0001 complete [ 1196.995647] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 23:52:42 (1788666762) [ 1212.601464] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1220.789510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.037451] LustreError: 47675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.055455] LustreError: 47675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 1225.133181] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1232.635149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.150736] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1239.427196] Lustre: 48827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1245.165359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.899860] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1259.967036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.163553] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1273.317714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1277.251870] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1289.185643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1289.712101] Lustre: server umount lustre-MDT0000 complete [ 1293.071017] LustreError: 48829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666860 with bad export cookie 11601853534770521866 [ 1293.074863] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.084299] LustreError: 48829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1293.479180] Lustre: server umount lustre-MDT0001 complete [ 1307.626427] Lustre: server umount lustre-OST0000 complete [ 1320.416820] Lustre: server umount lustre-OST0001 complete [ 1330.787269] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 23:54:56 (1788666896) [ 1333.483219] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1335.355675] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 23:55:01 (1788666901) [ 1345.326099] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 1354.204506] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1374.612924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.804605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1374.863104] Lustre: lustre-MDT0000: new disk, initializing [ 1374.921271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1378.852555] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1387.305095] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1392.812637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.068226] Lustre: lustre-OST0000: new disk, initializing [ 1393.070809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1393.078889] Lustre: Skipped 1 previous similar message [ 1393.085340] Lustre: 54904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1394.604669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1394.620817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1394.644848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1397.633849] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1405.669428] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1407.651866] Lustre: DEBUG MARKER: create test files [ 1435.619724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.632879] Lustre: Skipped 14 previous similar messages [ 1435.640292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.646269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.652422] Lustre: Skipped 20 previous similar messages [ 1447.392392] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1447.658689] Lustre: server umount lustre-OST0000 complete [ 1451.723285] Lustre: server umount lustre-MDT0000 complete [ 1461.864173] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1464.426619] Key type lgssc unregistered [ 1464.672701] LNet: 56530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.679896] LNetError: 56530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.692866] LNet: Removed LNI 192.168.206.133@tcp [ 1465.466142] Key type .llcrypt unregistered [ 1465.469319] Key type ._llcrypt unregistered [ 1490.315830] Key type ._llcrypt registered [ 1490.318203] Key type .llcrypt registered [ 1490.400975] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1491.211996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.235943] alg: No test for adler32 (adler32-zlib) [ 1492.263729] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1492.469720] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1494.089641] Key type lgssc registered [ 1495.067538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1502.136913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1502.162046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.637058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.615613] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1514.794581] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1520.994090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1521.298990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.625942] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1528.516538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:98 to 0x240000400:129) [ 1528.586263] Lustre: 59237:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1528.596545] Lustre: 59237:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1528.602445] Lustre: 59237:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1528.606957] Lustre: 59237:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1528.611784] Lustre: 59237:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1528.623474] Lustre: 59237:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1533.974676] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1538.873659] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1543.737329] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1580.002052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1580.008957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.027424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.196989] Lustre: server umount lustre-OST0000 complete [ 1585.862394] Lustre: server umount lustre-MDT0000 complete [ 1596.018724] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1598.439205] Key type lgssc unregistered [ 1598.652239] LNet: 60868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.659592] LNetError: 60868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.673210] LNet: Removed LNI 192.168.206.133@tcp [ 1599.473177] Key type .llcrypt unregistered [ 1599.474746] Key type ._llcrypt unregistered [ 1628.812147] Key type ._llcrypt registered [ 1628.814851] Key type .llcrypt registered [ 1628.927174] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1629.615159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1629.834077] alg: No test for adler32 (adler32-zlib) [ 1630.804541] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1631.044372] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1632.713815] Key type lgssc registered [ 1633.826842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1641.045827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1641.082451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.604055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.889807] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1653.657648] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1660.706874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1661.083405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.256745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1667.258726] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1667.339735] Lustre: 63704:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1667.347397] Lustre: 63704:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1667.357285] Lustre: 63704:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1667.374110] Lustre: 63704:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1667.383893] Lustre: 63704:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1667.393787] Lustre: 63704:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1674.462316] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.336336] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1688.035252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.039545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.066992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.162505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.800726] Lustre: server umount lustre-OST0000 complete [ 1698.041751] Lustre: server umount lustre-MDT0000 complete [ 1709.625729] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1712.167517] Key type lgssc unregistered [ 1712.403906] LNet: 65270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.413561] LNetError: 65270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.436406] LNet: Removed LNI 192.168.206.133@tcp [ 1713.191377] Key type .llcrypt unregistered [ 1713.196756] Key type ._llcrypt unregistered [ 1729.984214] Key type ._llcrypt registered [ 1729.991057] Key type .llcrypt registered [ 1730.093202] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 1745.075497] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1746.311880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.336238] alg: No test for adler32 (adler32-zlib) [ 1747.375921] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1747.648534] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1749.360210] Key type lgssc registered [ 1750.249169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.707392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.724808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.973665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1784.001307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1784.108810] Lustre: lustre-MDT0000: new disk, initializing [ 1784.188661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.215751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1788.588186] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1798.350038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.486991] Lustre: 68416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1798.557220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1798.562381] Lustre: Skipped 1 previous similar message [ 1798.650644] Lustre: lustre-MDT0001: new disk, initializing [ 1798.753927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.778441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1798.795784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1802.594657] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1810.736927] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1815.301570] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1820.683738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.959679] Lustre: lustre-OST0000: new disk, initializing [ 1820.962932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1820.967891] Lustre: 70380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1821.050372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.085853] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1827.362501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1827.373276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1827.507349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1836.173542] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1840.580482] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1840.794610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1844.850674] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1845.023193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1848.295478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.301259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.307202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.312693] Lustre: Skipped 1 previous similar message [ 1852.901512] Lustre: server umount lustre-OST0000 complete [ 1858.536387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.541822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.559025] Lustre: Skipped 1 previous similar message [ 1858.570053] Lustre: Skipped 2 previous similar messages [ 1862.690982] Lustre: server umount lustre-MDT0000 complete [ 1866.244568] LustreError: 68408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667434 with bad export cookie 9331560765647659123 [ 1866.247311] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1866.258982] LustreError: 68408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1866.562068] Lustre: server umount lustre-MDT0001 complete [ 1874.614568] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 00:04:00 (1788667440) [ 1885.137773] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1923.136777] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 1933.399449] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1971.596366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.814309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1971.882826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1972.024974] Lustre: lustre-MDT0000: new disk, initializing [ 1972.144928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.158309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1976.426166] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1987.052068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.148499] Lustre: 76403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1987.197696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1987.202590] Lustre: Skipped 1 previous similar message [ 1987.296817] Lustre: lustre-MDT0001: new disk, initializing [ 1987.379063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.414105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1987.433558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1991.598846] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1999.712646] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2004.892559] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.964980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.206857] Lustre: lustre-OST0000: new disk, initializing [ 2011.210496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2011.217796] Lustre: 78370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2011.299687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.144891] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2019.401652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2019.412413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2019.514624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2026.834476] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2031.296441] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2031.454815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2035.666418] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2035.928152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2040.290172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2040.297033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.311873] LustreError: Skipped 2 previous similar messages [ 2040.316566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2040.325533] Lustre: Skipped 1 previous similar message [ 2045.409066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.423738] Lustre: Skipped 2 previous similar messages [ 2050.535101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.543907] Lustre: Skipped 1 previous similar message [ 2052.064132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2052.245270] Lustre: server umount lustre-OST0000 complete [ 2059.233828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2059.247667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.266847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.421992] Lustre: server umount lustre-MDT0000 complete [ 2065.889025] LustreError: 77276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.910407] LustreError: 77276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2066.510960] LustreError: 79911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667634 with bad export cookie 9331560765647659998 [ 2066.514443] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.527989] LustreError: 79911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2066.824114] Lustre: server umount lustre-MDT0001 complete [ 2075.160806] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 00:07:20 (1788667640) [ 2082.213911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.609765] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.692829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.136603] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2087.908716] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.034497] LustreError: 80784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.523595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.719869] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2109.432523] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2115.492488] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2122.247547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.564909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.575773] Lustre: Skipped 1 previous similar message [ 2128.537873] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2134.670144] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2137.819187] Lustre: 83455:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 8fe50315-d723-4dc6-82a4-463e02d9fae6 at adminstrative request [ 2138.089837] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.102618] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2148.326668] Lustre: MGS: Client 8fe50315-d723-4dc6-82a4-463e02d9fae6 (at 0@lo) reconnecting [ 2148.336851] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x81805d135c66a757 to 0x81805d135c66a9f0 [ 2148.358023] Lustre: MGC192.168.206.133@tcp: Connection restored to 0@lo (at 0@lo) [ 2174.404128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2174.566448] Lustre: lustre-OST0001: new disk, initializing [ 2174.569973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2174.574973] Lustre: 83848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2180.097583] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2180.183549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2180.194797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2180.324633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2188.054533] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2190.831212] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2190.831629] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.836192] LustreError: Skipped 1 previous similar message [ 2190.839347] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2190.846064] Lustre: Skipped 2 previous similar messages [ 2190.858247] Lustre: Skipped 3 previous similar messages [ 2205.152689] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2205.234509] Lustre: server umount lustre-OST0001 complete [ 2206.198927] LustreError: 82604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.208640] LustreError: 82604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2209.761451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2209.767882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.301376] Lustre: server umount lustre-OST0000 complete [ 2218.977342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2223.633409] Lustre: server umount lustre-MDT0000 complete [ 2226.664444] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.691068] LustreError: 80785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2227.446071] LustreError: 83847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667795 with bad export cookie 9331560765647661552 [ 2227.447718] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2227.466274] LustreError: 83847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2227.735190] Lustre: server umount lustre-MDT0001 complete [ 2235.618892] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 00:10:01 (1788667801) [ 2237.491836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2239.505309] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 00:10:05 (1788667805) [ 2241.265542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2243.358679] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 00:10:08 (1788667808) [ 2245.064996] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2247.273422] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 00:10:12 (1788667812) [ 2249.183564] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2251.221164] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 00:10:16 (1788667816) [ 2253.500263] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2259.919772] loop: module loaded [ 2259.939767] loop0: detected capacity change from 0 to 4294967296 [ 2259.989078] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2261.538798] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2264.719956] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2268.386690] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2293.440544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.503956] LustreError: 87973:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2293.511569] LustreError: 87973:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2293.523160] LustreError: 87973:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2293.539137] LustreError: 87973:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2312.537791] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 2321.946892] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2361.931776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.143581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2362.164220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2362.238894] Lustre: lustre-MDT0000: new disk, initializing [ 2362.303165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.313324] Lustre: Skipped 1 previous similar message [ 2362.339660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2366.906252] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2376.055367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.231290] Lustre: 90841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2376.511855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2380.800827] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2389.318441] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.992659] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2401.180104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2401.456720] Lustre: 92810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2407.339729] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2408.005099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2408.162490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2415.910308] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2420.062273] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2420.271606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2425.150026] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2425.462539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2428.904264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.907376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.917134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2428.917142] Lustre: Skipped 10 previous similar messages [ 2428.953709] Lustre: Skipped 4 previous similar messages [ 2433.792941] Lustre: server umount lustre-OST0000 complete [ 2443.905490] Lustre: server umount lustre-MDT0000 complete [ 2444.258022] LustreError: 90852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.287576] LustreError: 90852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2447.500667] LustreError: 90831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668015 with bad export cookie 9331560765647662168 [ 2447.512163] LustreError: 90831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2447.518802] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.017877] Lustre: server umount lustre-MDT0001 complete [ 2456.856316] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 00:13:42 (1788668022) [ 2464.030828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.327777] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2478.534666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.994849] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2490.810549] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.466725] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2504.718643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.063282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.070053] Lustre: Skipped 4 previous similar messages [ 2510.993204] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2518.733263] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2524.155580] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2524.341785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2529.500563] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2529.883762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2538.160030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.485390] Lustre: lustre-OST0001: new disk, initializing [ 2538.492259] Lustre: Skipped 2 previous similar messages [ 2538.497165] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2538.509435] Lustre: Skipped 3 previous similar messages [ 2538.533124] Lustre: 98590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2546.847546] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2547.867139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2547.885286] Lustre: Skipped 2 previous similar messages [ 2547.900591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2548.042966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2558.302597] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2564.065723] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2564.400533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2569.739463] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2570.046402] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2590.973708] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2617.105126] Lustre: Failing over lustre-MDT0000 [ 2617.329641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2617.344792] LustreError: Skipped 1 previous similar message [ 2617.353130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.374439] Lustre: Skipped 2 previous similar messages [ 2617.400970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.410789] Lustre: Skipped 4 previous similar messages [ 2617.755376] Lustre: server umount lustre-MDT0000 complete [ 2618.746714] LustreError: 96217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.764605] LustreError: 96217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2635.121607] Lustre: 66641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668187/real 1788668187] req@00000000d86a0472 x1875553618767616/t0(0) o400->MGC192.168.206.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788668203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2635.146561] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.541829] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2643.545634] LDISKFS-fs (dm-0): recovery complete [ 2643.551578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.476951] LustreError: 66638:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000abbd6a34 x1875553618774656/t0(0) o250->MGC192.168.206.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2645.791755] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2650.459412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2650.461931] Lustre: 96217:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2650.484220] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2651.119628] Lustre: 95224:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2651.130658] Lustre: 95224:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2651.170049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2651.240502] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2651.576060] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2651.899505] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2651.922115] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2652.264150] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2652.276634] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2652.291561] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2652.305123] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2652.955053] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2652.981609] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2653.312162] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2653.326105] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2654.599608] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2654.618457] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2655.267438] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2655.282356] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2655.600150] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2655.614202] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2658.811468] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2658.825022] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2659.470527] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2659.485751] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2659.809483] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2659.814766] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2666.868861] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2666.886443] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2667.528436] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2667.539792] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2667.880200] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2667.891215] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2682.887774] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2682.908604] Lustre: 101439:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 49 previous similar messages [ 2683.540126] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2683.549085] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2684.200792] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2684.215030] LustreError: 101439:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2710.265378] Lustre: 101439:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2710.275213] Lustre: 101439:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2710.283348] Lustre: 101439:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client de1cd27e-8169-4480-8180-f2a83b0a5e84@192.168.206.33@tcp [ 2710.299303] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2710.310355] Lustre: 101439:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000fb8711be x1875553601788928/t0(8589934781) o101->de1cd27e-8169-4480-8180-f2a83b0a5e84@192.168.206.33@tcp:53/0 lens 592/0 e 0 to 0 dl 1788668288 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2710.335377] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2710.357634] Lustre: lustre-MDT0000: Denying connection for new client de1cd27e-8169-4480-8180-f2a83b0a5e84 (at 192.168.206.33@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2710.373389] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2710.430858] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2710.439364] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2710.440468] Lustre: Skipped 2 previous similar messages [ 2710.492117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2710.494778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2727.914655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.938133] Lustre: Skipped 4 previous similar messages [ 2742.241989] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2742.381282] Lustre: server umount lustre-OST0000 complete [ 2743.273383] LustreError: 97042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.297981] LustreError: 97042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 2747.364411] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2747.374483] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2747.386265] Lustre: Skipped 6 previous similar messages [ 2760.163598] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2760.391273] Lustre: server umount lustre-OST0001 complete [ 2770.944973] Lustre: server umount lustre-MDT0000 complete [ 2775.339247] LustreError: 102526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668343 with bad export cookie 9331560765647713394 [ 2775.348775] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.361796] LustreError: 102526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2775.823098] Lustre: server umount lustre-MDT0001 complete [ 2786.684905] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 00:19:12 (1788668352) [ 2795.646899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.658960] Lustre: *** cfs_fail_loc=197, val=0*** [ 2811.232638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2816.352632] LustreError: 103431:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.133@tcp: failed processing log, type 4: rc = -110 [ 2821.562764] Lustre: *** cfs_fail_loc=197, val=0*** [ 2821.564730] Lustre: Skipped 65 previous similar messages [ 2842.145355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.158200] Lustre: Skipped 2 previous similar messages [ 2848.576763] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2855.198733] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2857.144960] Lustre: server umount lustre-OST0000 complete [ 2865.894224] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 00:20:31 (1788668431) [ 2887.476088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.017241] LustreError: 105051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.036180] LustreError: 105051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2893.163333] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2902.128511] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2905.896573] LustreError: 105082:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2905.904951] LustreError: 105082:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2905.909826] LustreError: 105082:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2906.196460] Lustre: server umount lustre-MDT0000 complete [ 2938.685482] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 2948.202586] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2983.772692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.986877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2984.011022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2984.067437] Lustre: lustre-MDT0000: new disk, initializing [ 2984.150756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2988.497907] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2998.673628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.787794] Lustre: 109255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2999.125795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3003.674923] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3014.071286] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3019.943685] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3026.699153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3027.077709] Lustre: 111226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3032.622242] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3034.237503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3041.502693] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3045.858450] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3046.059889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3051.260688] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3051.483351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3054.562503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.574673] LustreError: Skipped 2 previous similar messages [ 3054.585070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.608554] Lustre: Skipped 4 previous similar messages [ 3054.620796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3054.631476] Lustre: Skipped 10 previous similar messages [ 3059.413393] Lustre: server umount lustre-OST0000 complete [ 3069.326366] Lustre: server umount lustre-MDT0000 complete [ 3072.950471] LustreError: 109249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668640 with bad export cookie 9331560765647724083 [ 3072.959808] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3073.262584] Lustre: server umount lustre-MDT0001 complete [ 3081.755050] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 00:24:07 (1788668647) [ 3091.430869] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3094.041342] Key type lgssc unregistered [ 3094.335843] LNet: 113711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3094.342500] LNetError: 113711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3094.369391] LNet: Removed LNI 192.168.206.133@tcp [ 3095.222269] Key type .llcrypt unregistered [ 3095.225781] Key type ._llcrypt unregistered [ 3112.223294] Key type ._llcrypt registered [ 3112.226186] Key type .llcrypt registered [ 3112.327625] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 3127.548701] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3128.482678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3128.618041] alg: No test for adler32 (adler32-zlib) [ 3129.755298] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3130.026782] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3131.722105] Key type lgssc registered [ 3132.778198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3171.563932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3171.598056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.826300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3172.858960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3172.930704] Lustre: lustre-MDT0000: new disk, initializing [ 3172.994359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.007625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3177.054404] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3185.487613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.681601] Lustre: lustre-OST0000: new disk, initializing [ 3185.686445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3185.691111] Lustre: Skipped 1 previous similar message [ 3185.696338] Lustre: 117878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3185.772807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.804061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3189.809227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3189.837769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3190.424327] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3199.192858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3199.309665] Lustre: lustre-OST0001: new disk, initializing [ 3199.312967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3199.317860] Lustre: 118741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3199.379840] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3204.732161] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3205.236088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3205.247490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3205.295583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3215.332174] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.335316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.354412] Lustre: Skipped 1 previous similar message [ 3219.213622] Lustre: server umount lustre-MDT0000 complete [ 3224.841612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3229.561991] LustreError: 118740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668797 with bad export cookie 2428383721278372803 [ 3229.577342] LustreError: 118740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3229.580238] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3229.666907] Lustre: server umount lustre-OST0000 complete [ 3233.439555] Lustre: server umount lustre-OST0001 complete [ 3241.320250] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 00:26:47 (1788668807) [ 3262.349377] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 00:27:08 (1788668828) [ 3270.742977] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3289.324458] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 00:27:35 (1788668855) [ 3323.283596] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 3332.136185] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3363.357134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.523713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3363.546919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3363.626535] Lustre: lustre-MDT0000: new disk, initializing [ 3363.692149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.705755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3367.361814] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3376.866161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.974451] Lustre: 126493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3377.042116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3377.047765] Lustre: Skipped 1 previous similar message [ 3377.115946] Lustre: lustre-MDT0001: new disk, initializing [ 3377.233784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3377.288620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3377.312819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3380.912764] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3389.124956] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3394.901717] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3402.016492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.396375] Lustre: lustre-OST0000: new disk, initializing [ 3402.401855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3402.416515] Lustre: 128461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3402.537066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.534852] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3412.055756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3412.068332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3412.206853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3417.162758] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3420.716205] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3420.878522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3425.320899] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3425.505873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3427.810620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3427.827090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.840179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3427.846981] Lustre: Skipped 2 previous similar messages [ 3432.931962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3432.942916] Lustre: Skipped 1 previous similar message [ 3433.484958] Lustre: server umount lustre-OST0000 complete [ 3438.568529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3438.579788] LustreError: Skipped 1 previous similar message [ 3438.589345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.607456] Lustre: Skipped 1 previous similar message [ 3438.612023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.174049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.175506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.385432] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.577625] Lustre: server umount lustre-MDT0000 complete [ 3456.626940] LustreError: 127287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669024 with bad export cookie 2428383721278374028 [ 3456.635205] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3456.642326] LustreError: 127287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3457.223497] Lustre: server umount lustre-MDT0001 complete [ 3465.617636] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 00:30:31 (1788669031) [ 3471.796354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.165750] LustreError: 130872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.188508] LustreError: 130872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3472.280741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.858409] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3477.485334] LustreError: 130873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.594395] LustreError: 130872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3486.849390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.594984] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3501.377883] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3507.132780] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3513.809619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3514.117340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.122630] Lustre: Skipped 1 previous similar message [ 3521.301627] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3529.224860] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3537.069123] Lustre: *** cfs_fail_loc=159, val=0*** [ 3553.661174] Lustre: lustre-MDT0000: Client 6f5bc863-7af1-4df3-87d8-8092157b3501 (at 192.168.206.33@tcp) reconnecting [ 3555.352517] Lustre: *** cfs_fail_loc=159, val=0*** [ 3555.358716] Lustre: Skipped 5 previous similar messages [ 3571.586077] Lustre: lustre-MDT0000: Client 6f5bc863-7af1-4df3-87d8-8092157b3501 (at 192.168.206.33@tcp) reconnecting [ 3577.970465] Lustre: server umount lustre-OST0000 complete [ 3583.970276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.974959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3583.992712] Lustre: Skipped 1 previous similar message [ 3584.011568] Lustre: Skipped 4 previous similar messages [ 3594.209813] Lustre: 115086:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669146/real 1788669146] req@00000000e7e6af22 x1875555067730688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788669162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3594.212943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.225336] Lustre: 115086:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3595.235562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3595.370835] Lustre: server umount lustre-MDT0000 complete [ 3598.631388] LustreError: 130852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669166 with bad export cookie 2428383721278374903 [ 3598.635687] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.644764] LustreError: 130852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3598.883759] Lustre: server umount lustre-MDT0001 complete [ 3608.531546] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3610.723717] Key type lgssc unregistered [ 3611.039823] LNet: 134648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3611.052542] LNetError: 134648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3611.078139] LNet: Removed LNI 192.168.206.133@tcp [ 3612.069525] Key type .llcrypt unregistered [ 3612.072664] Key type ._llcrypt unregistered [ 3636.331853] Key type ._llcrypt registered [ 3636.339452] Key type .llcrypt registered [ 3636.528260] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3637.257751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3637.362356] alg: No test for adler32 (adler32-zlib) [ 3638.524419] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3638.814736] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3640.568358] Key type lgssc registered [ 3641.612369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3648.379808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3648.395391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.711601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.048317] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3659.242468] LustreError: 136059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.268746] LustreError: 136059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3660.547117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3661.146806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3666.751888] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3675.234462] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3681.548060] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3687.917753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.257878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.970504] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3694.392665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3700.722817] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3713.092378] Lustre: *** cfs_fail_loc=159, val=0*** [ 3729.304029] Lustre: lustre-MDT0000: Client 4d95f367-3ff9-4c74-be11-92dea399d65e (at 192.168.206.33@tcp) reconnecting [ 3731.540520] Lustre: *** cfs_fail_loc=159, val=0*** [ 3731.552420] Lustre: Skipped 2 previous similar messages [ 3747.199246] Lustre: lustre-MDT0001: Client 4d95f367-3ff9-4c74-be11-92dea399d65e (at 192.168.206.33@tcp) reconnecting [ 3748.586426] Lustre: *** cfs_fail_loc=159, val=0*** [ 3748.589793] Lustre: Skipped 1 previous similar message [ 3764.168133] Lustre: lustre-MDT0001: Client 4d95f367-3ff9-4c74-be11-92dea399d65e (at 192.168.206.33@tcp) reconnecting [ 3780.646900] Lustre: 139122:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3783.479823] Lustre: *** cfs_fail_loc=159, val=0*** [ 3783.490191] Lustre: Skipped 3 previous similar messages [ 3798.930701] Lustre: lustre-MDT0000: Client 20288ea6-f72e-4024-99b4-95251c41be6a (at 192.168.206.33@tcp) reconnecting [ 3800.429382] Lustre: *** cfs_fail_loc=159, val=0*** [ 3800.432649] Lustre: Skipped 14 previous similar messages [ 3815.826180] Lustre: lustre-MDT0000: Client 20288ea6-f72e-4024-99b4-95251c41be6a (at 192.168.206.33@tcp) reconnecting [ 3817.340355] Lustre: 139409:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3823.084759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3823.092196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.105756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.713510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.726545] Lustre: Skipped 2 previous similar messages [ 3828.937530] Lustre: server umount lustre-OST0000 complete [ 3833.828581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.844902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.849566] Lustre: Skipped 2 previous similar messages [ 3838.946140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.958972] Lustre: Skipped 2 previous similar messages [ 3839.205689] Lustre: server umount lustre-MDT0000 complete [ 3842.478051] LustreError: 136041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669410 with bad export cookie 16598932077548732216 [ 3842.480041] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.484754] LustreError: 136041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3842.761017] Lustre: server umount lustre-MDT0001 complete [ 3851.531739] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3853.684851] Key type lgssc unregistered [ 3853.917008] LNet: 140460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3853.925170] LNetError: 140460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3854.953393] LNet: Removed LNI 192.168.206.133@tcp [ 3855.649473] Key type .llcrypt unregistered [ 3855.651527] Key type ._llcrypt unregistered [ 3874.364890] Key type ._llcrypt registered [ 3874.367425] Key type .llcrypt registered [ 3874.479882] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3875.094406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3875.120911] alg: No test for adler32 (adler32-zlib) [ 3876.188965] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3876.411536] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3878.090108] Key type lgssc registered [ 3879.194386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3885.500927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3885.513360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.838619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.303284] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3896.293102] LustreError: 141849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.309463] LustreError: 141849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3897.696648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.069171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3901.832656] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3908.984618] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3913.724725] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3920.330856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3920.663190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.736223] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3926.266346] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3931.966355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3933.631549] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3938.502231] Lustre: 144433:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3941.289377] Lustre: 144482:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3942.588363] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3943.631122] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3947.504959] Lustre: 144629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3948.420067] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3949.434532] Lustre: 144727:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3957.730839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.735818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.740113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.759032] Lustre: Skipped 1 previous similar message [ 3959.924411] Lustre: server umount lustre-OST0000 complete [ 3964.897587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.902472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.922970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3964.933405] Lustre: Skipped 1 previous similar message [ 3967.970251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.970376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3967.983632] Lustre: Skipped 1 previous similar message [ 3969.733767] Lustre: server umount lustre-MDT0000 complete [ 3973.090475] LustreError: 141849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.108626] LustreError: 141849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3973.179941] LustreError: 141831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669541 with bad export cookie 13330887938850856515 [ 3973.181283] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3973.197717] LustreError: 141831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3973.476185] Lustre: server umount lustre-MDT0001 complete [ 3986.085929] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3988.957875] Key type lgssc unregistered [ 3989.174351] LNet: 145789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3989.183890] LNetError: 145789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3990.250056] LNet: Removed LNI 192.168.206.133@tcp [ 3991.167075] Key type .llcrypt unregistered [ 3991.170990] Key type ._llcrypt unregistered [ 4014.566177] Key type ._llcrypt registered [ 4014.568643] Key type .llcrypt registered [ 4014.694740] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4015.961272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4016.021829] alg: No test for adler32 (adler32-zlib) [ 4017.054463] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4017.289104] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4018.960310] Key type lgssc registered [ 4020.148912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4028.767443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4028.788254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.158461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.686369] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4040.678994] LustreError: 147199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.720295] LustreError: 147199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4045.793454] LustreError: 147198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.321849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.802452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4050.716826] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4057.350087] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4062.101847] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4068.237653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4068.573621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.661559] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4073.846740] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4075.767569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4080.094556] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4086.192688] Lustre: *** cfs_fail_loc=159, val=0*** [ 4102.545434] Lustre: lustre-MDT0000: Client 624973ad-9319-4361-9216-b7d3c86e960f (at 192.168.206.33@tcp) reconnecting [ 4107.749376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4107.762346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.765088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4107.780884] LustreError: Skipped 6 previous similar messages [ 4109.342436] Lustre: server umount lustre-OST0000 complete [ 4113.379879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4113.392714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.414031] Lustre: Skipped 1 previous similar message [ 4113.425917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4113.435539] Lustre: Skipped 1 previous similar message [ 4116.965839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.967950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4117.003582] Lustre: Skipped 1 previous similar message [ 4119.218450] Lustre: server umount lustre-MDT0000 complete [ 4122.081871] LustreError: 147195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4122.093691] LustreError: 147195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4122.721250] LustreError: 149052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669690 with bad export cookie 11794472260083670536 [ 4122.724218] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4122.730158] LustreError: 149052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4123.150072] Lustre: server umount lustre-MDT0001 complete [ 4133.152533] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 4135.710278] Key type lgssc unregistered [ 4135.942941] LNet: 150859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4135.950845] LNetError: 150859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4135.961390] LNet: Removed LNI 192.168.206.133@tcp [ 4136.750353] Key type .llcrypt unregistered [ 4136.752196] Key type ._llcrypt unregistered [ 4158.237298] Key type ._llcrypt registered [ 4158.243867] Key type .llcrypt registered [ 4158.325460] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4159.318803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4159.411737] alg: No test for adler32 (adler32-zlib) [ 4160.433823] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4160.671381] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4162.312173] Key type lgssc registered [ 4163.159153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4169.833041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4169.848989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.314925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.105395] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4181.477656] LustreError: 152271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.508061] LustreError: 152271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4182.279870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.642914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4186.580823] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4192.551071] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4197.157347] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4202.711384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.110875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.215216] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4208.370373] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4215.065029] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4216.439416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4237.379393] Lustre: lustre-OST0000: evicting 54358c54-8255-4d3d-a835-b711a0cc25d4 (at 192.168.206.33@tcp) by administrative request [ 4237.390131] Lustre: Skipped 1 previous similar message [ 4268.001312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4268.005364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.010654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4268.028259] Lustre: Skipped 1 previous similar message [ 4273.123865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4273.128937] Lustre: Skipped 2 previous similar messages [ 4278.242157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4278.250600] Lustre: Skipped 1 previous similar message [ 4279.265105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4279.414953] Lustre: server umount lustre-OST0000 complete [ 4283.367873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.375535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.388332] Lustre: Skipped 1 previous similar message [ 4283.398675] Lustre: Skipped 1 previous similar message [ 4288.481498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4288.497762] Lustre: Skipped 1 previous similar message [ 4288.781950] Lustre: server umount lustre-MDT0000 complete [ 4291.887563] LustreError: 152253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669859 with bad export cookie 8967599083643860366 [ 4291.890972] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.901629] LustreError: 152253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4292.216124] Lustre: server umount lustre-MDT0001 complete [ 4301.220225] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 4303.404575] Key type lgssc unregistered [ 4303.728067] LNet: 156339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4303.738164] LNetError: 156339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4303.756406] LNet: Removed LNI 192.168.206.133@tcp [ 4304.453145] Key type .llcrypt unregistered [ 4304.455403] Key type ._llcrypt unregistered [ 4375.295314] Key type ._llcrypt registered [ 4375.297612] Key type .llcrypt registered [ 4375.382757] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4376.324731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4376.479055] alg: No test for adler32 (adler32-zlib) [ 4377.600843] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4377.842390] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4379.544123] Key type lgssc registered [ 4380.682723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4388.845461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4388.872456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.538350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.513932] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4400.104435] LustreError: 158823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.136628] LustreError: 158823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4404.440853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.800248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4409.086366] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4414.910359] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4419.696950] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4425.830583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.337337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.431587] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4433.504368] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4435.569309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4440.430049] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4450.274881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4450.281370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.292086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4450.476202] Lustre: server umount lustre-OST0000 complete [ 4455.905402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.910905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.919839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.538369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.547053] Lustre: Skipped 3 previous similar messages [ 4463.639655] Lustre: server umount lustre-MDT0000 complete [ 4466.658079] LustreError: 161088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.684343] LustreError: 161088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4466.709859] LustreError: 158803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670034 with bad export cookie 11916522203720757679 [ 4466.713841] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4466.719295] LustreError: 158803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4467.004318] Lustre: server umount lustre-MDT0001 complete [ 4475.841600] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 4478.078889] Key type lgssc unregistered [ 4478.316850] LNet: 162366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4478.333135] LNetError: 162366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4478.349145] LNet: Removed LNI 192.168.206.133@tcp [ 4478.964831] Key type .llcrypt unregistered [ 4478.968193] Key type ._llcrypt unregistered [ 4499.370483] Key type ._llcrypt registered [ 4499.373366] Key type .llcrypt registered [ 4499.492480] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4500.297984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4500.471546] alg: No test for adler32 (adler32-zlib) [ 4501.592613] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4501.920907] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4503.648350] Key type lgssc registered [ 4504.968701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4521.144409] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 00:48:06 (1788670086) [ 4537.450753] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 4546.699979] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4578.360252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4578.381794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.568157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4579.591954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4579.649921] Lustre: lustre-MDT0000: new disk, initializing [ 4579.715243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4579.734501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4583.070183] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4591.179483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.291747] Lustre: 166604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4591.370513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4591.378289] Lustre: Skipped 1 previous similar message [ 4591.456875] Lustre: lustre-MDT0001: new disk, initializing [ 4591.552661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4591.575306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4591.587447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4595.262099] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4602.770858] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4606.938102] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4612.925134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.184521] Lustre: lustre-OST0000: new disk, initializing [ 4613.188314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4613.199745] Lustre: 168573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4613.299985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4618.092449] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4623.456639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4623.467911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4623.610740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4626.625846] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4639.203079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.222330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.224968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.241117] Lustre: Skipped 1 previous similar message [ 4639.249567] Lustre: Skipped 1 previous similar message [ 4641.789735] Lustre: server umount lustre-OST0000 complete [ 4647.908969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4647.918260] LustreError: Skipped 1 previous similar message [ 4647.922568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.931036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.441776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.442994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.283229] Lustre: server umount lustre-MDT0000 complete [ 4654.916749] LustreError: 168572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670222 with bad export cookie 3054447589919370301 [ 4654.920790] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.925856] LustreError: 168572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4655.227843] Lustre: server umount lustre-MDT0001 complete [ 4664.836974] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 4667.430571] Key type lgssc unregistered [ 4667.654705] LNet: 170476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4667.688665] LNetError: 170476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4667.727589] LNet: Removed LNI 192.168.206.133@tcp [ 4668.498872] Key type .llcrypt unregistered [ 4668.502228] Key type ._llcrypt unregistered [ 4692.031540] Key type ._llcrypt registered [ 4692.034541] Key type .llcrypt registered [ 4692.169257] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4692.873858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4693.021277] alg: No test for adler32 (adler32-zlib) [ 4694.040927] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4694.293247] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4696.096250] Key type lgssc registered [ 4696.981321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4703.979745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4703.996038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.321132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.113112] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4715.492629] LustreError: 171886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.510075] LustreError: 171886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4715.740571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.195623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4719.878206] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4726.173341] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4731.782992] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4738.023655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4738.246103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4743.508478] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4750.998574] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4752.562471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4761.813417] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4761.825806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.837277] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4761.849794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.864462] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4762.798050] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4762.813823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.822705] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4762.833831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4762.852409] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4763.654424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.609503] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4764.616141] Lustre: Skipped 1 previous similar message [ 4764.639981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.662634] Lustre: Skipped 1 previous similar message [ 4764.667516] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4764.672233] Lustre: Skipped 1 previous similar message [ 4764.678856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.701757] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4764.715094] Lustre: Skipped 1 previous similar message [ 4765.429090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.141812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.736952] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4766.741681] Lustre: Skipped 2 previous similar messages [ 4766.755913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.762893] Lustre: Skipped 2 previous similar messages [ 4766.765769] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4766.769380] Lustre: Skipped 2 previous similar messages [ 4766.772689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.790177] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4766.795337] Lustre: Skipped 2 previous similar messages [ 4767.543282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.314368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.043691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.709132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.489764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.189633] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4771.205603] Lustre: Skipped 5 previous similar messages [ 4771.218603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.233840] Lustre: Skipped 5 previous similar messages [ 4771.242234] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4771.249590] Lustre: Skipped 5 previous similar messages [ 4771.255873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.269911] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4771.275385] Lustre: Skipped 5 previous similar messages [ 4772.032335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.897889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.667442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.428408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.165418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.817204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.568947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.244838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.864741] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4777.875514] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4777.879693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.572065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.296974] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4779.304320] Lustre: Skipped 10 previous similar messages [ 4779.316584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.326746] Lustre: Skipped 10 previous similar messages [ 4779.329584] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4779.334835] Lustre: Skipped 10 previous similar messages [ 4779.338486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.348107] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4779.353420] Lustre: Skipped 10 previous similar messages [ 4780.002672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.747864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.573334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.314647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.031784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.820159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.736462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.716384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.546682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.258841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.060683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.948543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.675086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.303286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.927548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.680817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.613764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.469676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.245274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.195924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.102730] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4796.106571] Lustre: Skipped 20 previous similar messages [ 4796.132773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.146989] Lustre: Skipped 20 previous similar messages [ 4796.150743] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4796.155262] Lustre: Skipped 20 previous similar messages [ 4796.158784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.171907] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4796.185785] Lustre: Skipped 20 previous similar messages [ 4796.934786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.773956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.539570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.461977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.200686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.065739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.798101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.640844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.323510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.089994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.863749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.627220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.363594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.125210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.888543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.624769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.311867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.955819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.652943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.348263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.118772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.973590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.646079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.517325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.181407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.946464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.731837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.854836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.628612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.313220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.959330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.629025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.363840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.125538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.889244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.667731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.340910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.094863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.762872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.573655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.329101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.108227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.764634] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4828.771303] Lustre: Skipped 42 previous similar messages [ 4828.781943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.791206] Lustre: Skipped 42 previous similar messages [ 4828.794076] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4828.799065] Lustre: Skipped 42 previous similar messages [ 4828.801903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.814263] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4828.818746] Lustre: Skipped 42 previous similar messages [ 4829.546824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.246517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.941091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.560785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.257580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.011271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.881323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.733764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.455076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.279871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.030368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.797532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.587046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.254426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.024070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.672239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.535354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.221356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.959578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.619071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.354828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.075486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.929115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.671322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.402365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.075709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.811266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.605382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.467808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.192170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.822429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.517300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.355630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.080683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.890376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.688958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.419400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.084691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.801250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.431923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.177501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.892234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.628318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.257968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.850284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.456328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.151608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.784754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.485123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.178588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.810852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.390329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.070437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.795163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.407534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.060893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.880189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.592261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.338575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.015805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.683452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.329285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.086809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.932096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.722325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.414407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.102235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.802466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.554614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.307833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.112057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.854145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.505695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.126814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.780542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4883.405976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.085268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.663838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.291060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.891528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.524785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.125891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.759770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.405525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.047535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.661811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.368050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.988854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.710336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.282743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.086447] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4893.091113] Lustre: Skipped 90 previous similar messages [ 4893.104339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.113896] Lustre: Skipped 90 previous similar messages [ 4893.122071] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4893.126801] Lustre: Skipped 90 previous similar messages [ 4893.130176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.143019] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4893.148525] Lustre: Skipped 90 previous similar messages [ 4893.780500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.466304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.076012] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.689498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.301469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.913149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4897.556182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.279289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.985198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4899.593660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.253309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.914242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.459515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.038563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.546461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.110313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.734648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.314113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.877855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.422050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.973343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4906.552536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.063557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.599737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.153632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.718623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4909.410469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.067395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.834369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.356664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.901648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.417292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.980735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.560293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.305437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.880705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.458190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.048133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.697730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.422787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.252889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4919.112193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4919.920653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.744173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.507593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4922.273278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.191595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.851348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.497618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.301984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.113386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.028311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.817589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.634056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.443692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.242033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.859513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.563380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.297212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.259590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.236443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.004757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.934703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.691266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.634938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.411910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.248521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.977213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.830820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.560081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.313302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.143873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.968814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.780985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.598655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.381412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.269351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.026426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.787888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.559397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.390540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.190537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.086688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.906637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.687278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.529967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.381510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.143218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.913548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.761634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.501983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.325026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.171367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.977855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.684986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.429194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.228713] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4963.241258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.244806] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4964.073104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.948651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.631852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.415623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.188194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.894608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.682177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.682100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.537877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.312626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.055312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.724237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.420491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.093111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.852939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.686373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.526787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.427357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.329560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.143866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.065860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.807594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.617872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.384159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.189202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.860472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.677843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.307902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.008446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.891223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.560843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.331280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.030414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.753266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.469341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.148959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.816545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.519722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.335742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.066737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.738115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.491825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.180106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.944233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.589189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.301305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.988590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.822081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.702597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.494333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.291350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.060303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.943816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.581199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.314771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.038819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.706650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.394131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.191914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.932882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.677995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.394212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.025774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.622726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.223901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.838596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.439629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.111220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.728124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.314946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.850873] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5015.858382] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5015.859488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.469619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.114090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.737294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.364272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.015743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.663500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.302138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.953950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.627981] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5021.634892] Lustre: Skipped 176 previous similar messages [ 5021.663024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.672953] Lustre: Skipped 176 previous similar messages [ 5021.676298] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5021.681423] Lustre: Skipped 176 previous similar messages [ 5021.684567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.693334] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5021.701166] Lustre: Skipped 176 previous similar messages [ 5022.397920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.971784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.566438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.253991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.963723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.588706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.263705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.050465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.725240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.343746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.932589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.584477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.131633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.734175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.380281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.989432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.560603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.139566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.706054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.357711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.023183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.787257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.587250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.416164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.213349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.044507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.817503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.597211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.295389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.015796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.720823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.469790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.261662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.865179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.609339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.290919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.062050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.735201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.444283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.136662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.965073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.723191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.439088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.155425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.904736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.591075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.375750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.017274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.716289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.450429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.262497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.066742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.917053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.654948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.307993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.950119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.715495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.372665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.111114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.792291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.458128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.225235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.985049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.579858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.243529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.014983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.789385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.501368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.155691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.807648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.645679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.479557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.177943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.097703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.069840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.933215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.953532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.616331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.787389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.935369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.815862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.929718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.865808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.992872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.866405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.841196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.619420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.627807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.428044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.405395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.070613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.724448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.488793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.145420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.911412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.676653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.642579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.554950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.306195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.225641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.038843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.777843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.632449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.512211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.334212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.236370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.045379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.995222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.901906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.987230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.051444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.803195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.777511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.012646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.325687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.441791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.639160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.741113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.792945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.080062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.311307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.475435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.017452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.334922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.728599] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5123.745072] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5123.768686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.200607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.399716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.665933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.963822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.076378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.200223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.545441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.850580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.043993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.997528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.939234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.839306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.666433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.449632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.428463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.428135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.525995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.526926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.438282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.251471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.103920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.979725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.930862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.723285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.586507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.478809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.317838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.166834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.166722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.081444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.921087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.852442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.772875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.876768] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5157.892227] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5157.900660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.625052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.255745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.057908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.877484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.671696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.531535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.351609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.189747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.027255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.082069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.819315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.613105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.521500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.296639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.172622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.067366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.889116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.692262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.340426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.279202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.240098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.162705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.915864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.748395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.507806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.352773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.116188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.950957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.717831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.432565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.108218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.947611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.604557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.339690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.206628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.097966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.786497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.545615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.314724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.063057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.782763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.563555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.324701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.068431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.786950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.468405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.261581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.004357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.764641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.542923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.353704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.140346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.911440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.627667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.390506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.339525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.263499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.012452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.807131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.679259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.482955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.373676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.140511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.859874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.681443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.620352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.453675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.228784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.129538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.128676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.965644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.761152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.806059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.955259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.824825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.627482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.446295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.256690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.041980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.777371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.502402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.292234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.119979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.847892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.557800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.215535] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5227.231095] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5227.232447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.949859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.727882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.392493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.209728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.992809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.831194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.502448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.295699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.073063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.839459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.556931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.251625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.973464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.723592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.567773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.465752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.317188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.235896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.052548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.978796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.720581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.515479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.336721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.028280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.977641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.715295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.591920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.348823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.211554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.194493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.216348] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5252.237359] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5252.241686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.137220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.990188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.811796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.635564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.582642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.467730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.349149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.207907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.995696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.786939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.559983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.287972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.148153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.989800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.706925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.575372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.332354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.139532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.941551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.755126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.542216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.317693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.950622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.802900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.535992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.378088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.089458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.937797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.787800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.614491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.541036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.407693] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5278.416904] Lustre: Skipped 307 previous similar messages [ 5278.426754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.438317] Lustre: Skipped 307 previous similar messages [ 5278.449275] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5278.457974] Lustre: Skipped 307 previous similar messages [ 5278.461720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.475859] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5278.483447] Lustre: Skipped 307 previous similar messages [ 5279.261168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.161504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.927837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.718872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.437099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.204475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.989877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.804500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.529809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.319280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.073191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.861321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.567696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.407795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.292967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.096940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.855455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.758994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.748052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.895435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.709045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.485874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.226676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.955555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.759895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.708028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.475962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.536146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.406396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.152836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.963880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.824130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.490050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.565156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.344199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.097588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.878573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.803181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.670461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.495698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.198961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.022412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.929864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.771214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.638154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.708515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.678611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.701253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.814092] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5319.825853] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5319.844924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.736878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.715925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.757068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.624441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.596404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.410669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.340311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.315195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.089580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.994681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.910530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.138661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.074313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.985723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.006534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.929817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.712247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.529697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.191800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.050212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.048109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.869389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.668692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.608031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.423985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.235500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.027449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.822657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.526164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.218801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.942507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.626871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.256297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.895060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.564128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.395255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.098597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.975779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.751575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.512560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.396964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.302049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.277172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.166580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.883841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.729094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.487730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.321069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.146860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.979137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.688162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.598416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.227067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.951388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.878215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.709748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.572830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.436407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.258463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.464953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.722574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.703351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.534890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.240683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.105284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.036459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.826228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.749307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.848367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.837655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.760923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.755208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.816424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.828123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.794154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.972884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.150662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.303793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.538160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.515454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.644452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.578434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.549254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.299381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.322775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.193619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.265208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.257214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.209264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.036040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.836789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.729324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.808092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.542300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.362475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.130758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.033899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.954589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.846203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.761227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.544191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.319135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.182387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.937412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.902041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.686518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.525473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.463091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.349609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.175875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.074103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.878699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.572177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.301602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.186553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.983168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.749132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.525583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.361267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.133145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.912263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.719039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.514873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.320201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.063450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.896477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.674869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.522907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.320133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.034434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.783854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.564461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.410853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.412609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.310159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.250350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.126808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.982712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.911290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.690648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.634032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.529125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.275291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.065706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.938809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.800262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.697357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.588546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.487418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.269123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.078214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.977501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.788978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.570227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.500300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.216902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.044618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.946079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.834550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.700763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.462597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.272991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.252873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.057191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.950736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.843433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.593980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.414556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.192283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.963455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.703295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.510906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.170722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.883463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.709153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.510895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.357616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.243837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.983857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.872930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.927285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.808677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.579602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.490176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.425732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.280066] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5480.287402] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 5480.294710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.298635] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5480.317799] LustreError: 171905:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 5481.266064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.059735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.830765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.711506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.584779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.304114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.012779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.738881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.450396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.218283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.966242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.720937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.338321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.063451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.754350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.419267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.077693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.749654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.347452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.004576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.815860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.601057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.230100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.885646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.598277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.323051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.990771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.609514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.166945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.896481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.495092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.396086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.213689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.035808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.867379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.757878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.799371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.745725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.561503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.327745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.954091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.699038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.362094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.026373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.708916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.419357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.113675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.801907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.424323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.127131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.732850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.555602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.309652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.071689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.776536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.534364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.234538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.928981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.659192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.306751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.055180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.751435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.472431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.240773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.997652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.794046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.549489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.310495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.136745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.959410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.640565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.434797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.151393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.168450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.080321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.932849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.792307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.585781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.369825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.245179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.934128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.738644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.305565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.959840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.598414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.416578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.051776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.756269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.489502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.131352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.745183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.410290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.069378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.761034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.523993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.323624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.078299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.738509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.474114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.260516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.021386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.806451] LustreError: 173706:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 5971.815625] LustreError: 171905:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6625.763040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.775482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.795921] Lustre: Skipped 336 previous similar messages [ 6625.809530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.786387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.802452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.802663] LustreError: Skipped 2 previous similar messages [ 6627.571114] Lustre: server umount lustre-OST0000 complete [ 6634.984258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.991759] Lustre: Skipped 1 previous similar message [ 6638.393457] Lustre: server umount lustre-MDT0000 complete [ 6640.099086] LustreError: 171885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.111294] LustreError: 171885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6642.633975] LustreError: 171867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672210 with bad export cookie 1015265108881981374 [ 6642.638909] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6642.657460] LustreError: 171867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6643.108683] Lustre: server umount lustre-MDT0001 complete [ 6653.380401] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 6657.082576] Key type lgssc unregistered [ 6657.462201] LNet: 227124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6657.485452] LNetError: 227124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6657.523129] LNet: Removed LNI 192.168.206.133@tcp [ 6658.801747] Key type .llcrypt unregistered [ 6658.808696] Key type ._llcrypt unregistered [ 6686.097392] Key type ._llcrypt registered [ 6686.100681] Key type .llcrypt registered [ 6686.208821] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 6687.062743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6687.103705] alg: No test for adler32 (adler32-zlib) [ 6688.121568] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6688.420307] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 6690.160204] Key type lgssc registered [ 6691.241852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6698.673167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6698.692215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.126767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6704.859771] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6710.246586] LustreError: 228536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.277870] LustreError: 228536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6713.989146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6714.333523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6718.984093] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6726.450410] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6732.201407] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6738.567689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6738.921763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.043336] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6746.152224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6752.551532] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.922417] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6759.266664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.274119] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6759.280370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6759.293510] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6759.296315] LustreError: 228555:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6761.448579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.465432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6761.470928] Lustre: Skipped 1 previous similar message [ 6765.448547] Lustre: lustre-OST0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 6766.567794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.578666] Lustre: Skipped 1 previous similar message [ 6770.525532] Lustre: lustre-OST0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 6775.265270] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6775.418092] Lustre: server umount lustre-OST0000 complete [ 6779.344223] LustreError: 228555:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6809.040389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6809.437518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.724576] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6825.634425] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6826.870586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6826.894732] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6826.899101] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6826.931037] Lustre: Skipped 1 previous similar message [ 6827.688991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6833.586527] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d23460b8000.ost_server_uuid 50 [ 6835.289933] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d23460b8000.ost_server_uuid in FULL state after 0 sec [ 6842.350439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.351690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.366272] Lustre: Skipped 1 previous similar message [ 6842.368174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.368181] Lustre: Skipped 2 previous similar messages [ 6846.915384] Lustre: server umount lustre-OST0000 complete [ 6852.577711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6852.578255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.601917] Lustre: Skipped 1 previous similar message [ 6852.610804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6852.616153] Lustre: Skipped 2 previous similar messages [ 6857.553041] Lustre: server umount lustre-MDT0000 complete [ 6857.703820] LustreError: 229060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.737622] LustreError: 229060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6861.477757] LustreError: 228515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672429 with bad export cookie 5370237265437373034 [ 6861.490585] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6861.491602] LustreError: 228515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6861.803333] Lustre: server umount lustre-MDT0001 complete [ 6873.154674] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 6875.606840] Key type lgssc unregistered [ 6875.847539] LNet: 233742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6875.855589] LNetError: 233742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6875.867728] LNet: Removed LNI 192.168.206.133@tcp [ 6876.648707] Key type .llcrypt unregistered [ 6876.651214] Key type ._llcrypt unregistered [ 6907.651548] Key type ._llcrypt registered [ 6907.654608] Key type .llcrypt registered [ 6907.750625] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 6908.425905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.483822] alg: No test for adler32 (adler32-zlib) [ 6909.517978] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6909.747262] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 6911.408175] Key type lgssc registered [ 6912.460362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6920.857761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6920.894778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.336298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6926.872651] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6932.453210] LustreError: 235469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.477660] LustreError: 235469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6936.910150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.333657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6942.020363] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6945.072325] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6951.970244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.404805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6958.915808] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6967.497826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6967.664445] Lustre: lustre-OST0001: new disk, initializing [ 6967.667281] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6967.672718] Lustre: 237778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6967.755233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6968.595743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6968.610774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6968.739863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6969.729980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6973.348778] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6984.249692] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6991.016557] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7005.668126] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.680310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7005.701330] Lustre: Skipped 2 previous similar messages [ 7005.716963] Lustre: Skipped 3 previous similar messages [ 7010.785742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7011.080317] Lustre: server umount lustre-MDT0000 complete [ 7015.910758] LustreError: 235464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.404284] LustreError: 235448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672584 with bad export cookie 8584790723625413313 [ 7016.428596] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.441714] LustreError: 235448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7016.940603] Lustre: server umount lustre-MDT0001 complete [ 7031.760400] Lustre: server umount lustre-OST0000 complete [ 7045.675602] Lustre: server umount lustre-OST0001 complete [ 7068.538251] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7077.867274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.231336] LustreError: 240887:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.252726] LustreError: 240887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7078.366218] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.285344] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7083.500396] LustreError: 240888:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.612896] LustreError: 240887:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.281064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.558241] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7094.248717] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7097.150159] Lustre: 242033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7103.253475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7104.686098] LustreError: 242390:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7104.697061] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7109.153881] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7115.263541] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7117.090844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7117.295306] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7117.300916] Lustre: Skipped 1 previous similar message [ 7122.834515] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7129.669693] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7132.806798] Lustre: 243902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7141.857334] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7141.864822] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.881805] Lustre: Skipped 1 previous similar message [ 7141.889380] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.893150] Lustre: Skipped 3 previous similar messages [ 7142.883518] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.897055] Lustre: Skipped 2 previous similar messages [ 7147.373609] Lustre: server umount mylustre-MDT0000 complete [ 7148.006971] LustreError: 240888:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.017282] LustreError: 240888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7150.564946] LustreError: 240864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672718 with bad export cookie 8584790723625416162 [ 7150.575753] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.582879] LustreError: 240864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7150.819345] Lustre: server umount mylustre-MDT0001 complete [ 7163.367508] Lustre: server umount mylustre-OST0000 complete [ 7176.400235] Lustre: server umount mylustre-OST0001 complete [ 7198.232817] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7206.602332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.846223] LustreError: 246210:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.909357] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7211.452214] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7219.442927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.014476] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7226.357907] Lustre: 247354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7231.943464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.287427] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7243.987514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.230944] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:161) [ 7245.302261] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 7249.123067] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7251.380021] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7261.017412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7263.879167] Lustre: 249226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7270.884084] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.893350] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.907963] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.910564] Lustre: Skipped 4 previous similar messages [ 7276.301205] Lustre: server umount tfs-MDT0000 complete [ 7277.028592] LustreError: 246211:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.055409] LustreError: 246211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7279.222270] LustreError: 246186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672847 with bad export cookie 8584790723625418115 [ 7279.232087] LustreError: 246186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7279.232764] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.531165] Lustre: server umount tfs-MDT0001 complete [ 7293.188469] Lustre: server umount tfs-OST0000 complete [ 7306.237606] Lustre: server umount tfs-OST0001 complete [ 7326.441282] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7337.045866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.620692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.632832] Lustre: Skipped 3 previous similar messages [ 7342.391731] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7350.552093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.737995] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7358.735190] Lustre: 252617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7366.191891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.184260] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7376.111124] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:193) [ 7382.551928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7384.780545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.224298] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7392.499682] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7400.366591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7404.046288] Lustre: 254488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7411.066367] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 01:36:16 (1788672976) [ 7428.079877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7428.088747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.104313] Lustre: Skipped 3 previous similar messages [ 7428.111871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.123250] Lustre: Skipped 4 previous similar messages [ 7436.259602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7436.271321] Lustre: Skipped 7 previous similar messages [ 7441.377492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7441.764578] Lustre: server umount lustre-MDT0000 complete [ 7445.379156] LustreError: 253799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673013 with bad export cookie 8584790723625420040 [ 7445.395688] LustreError: 253799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7445.395957] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.499243] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.503964] LustreError: 251471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7446.518059] Lustre: Skipped 4 previous similar messages [ 7446.543470] LustreError: 251471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7451.867102] Lustre: server umount lustre-MDT0001 complete [ 7461.876253] Lustre: server umount lustre-OST0000 complete [ 7472.332373] Lustre: server umount lustre-OST0001 complete [ 7480.347472] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 7491.169297] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7528.756919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.946598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7528.973221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7529.032472] Lustre: lustre-MDT0000: new disk, initializing [ 7529.087126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7529.093721] Lustre: Skipped 3 previous similar messages [ 7529.119466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7533.295904] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7543.761253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7543.927056] Lustre: 257671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7544.045025] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7544.055510] Lustre: Skipped 1 previous similar message [ 7544.151206] Lustre: lustre-MDT0001: new disk, initializing [ 7544.281706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7544.296651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7548.271641] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7560.220788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.466399] Lustre: lustre-OST0000: new disk, initializing [ 7560.471938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7560.479342] Lustre: 259253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7562.269465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7562.294901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7562.400560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7568.802217] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7580.595777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7580.790886] Lustre: lustre-OST0001: new disk, initializing [ 7580.795994] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7580.801454] Lustre: 260126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7582.656428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7582.675246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7582.742696] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7587.543675] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7599.618096] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 01:39:25 (1788673165) [ 7611.824088] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 01:39:37 (1788673177) [ 7628.782565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.791933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7628.793593] Lustre: Skipped 1 previous similar message [ 7628.804172] Lustre: Skipped 9 previous similar messages [ 7632.456540] Lustre: server umount lustre-OST0000 complete [ 7636.450820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7642.694850] Lustre: server umount lustre-MDT0000 complete [ 7646.534423] LustreError: 257663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673214 with bad export cookie 8584790723625421818 [ 7646.537321] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7646.542952] LustreError: 257663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7652.945080] Lustre: server umount lustre-MDT0001 complete [ 7663.801859] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 7743.297172] Lustre: server umount lustre-OST0001 complete [ 7750.544334] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 7765.027575] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7803.262424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.565208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7803.587710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7803.671431] Lustre: lustre-MDT0000: new disk, initializing [ 7803.774141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7803.780428] Lustre: Skipped 3 previous similar messages [ 7803.804702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7808.368893] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7819.097430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.235962] Lustre: 264977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7819.545592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7824.042264] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7833.882417] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7840.042378] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7847.503793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7847.756801] Lustre: lustre-OST0000: new disk, initializing [ 7847.759802] Lustre: Skipped 1 previous similar message [ 7847.762944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7847.767234] Lustre: Skipped 2 previous similar messages [ 7847.774356] Lustre: 266944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7848.902679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7848.911607] Lustre: Skipped 1 previous similar message [ 7848.919609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7849.075155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7853.638516] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7863.614466] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7874.575986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.577372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7874.582521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7874.586626] Lustre: Skipped 5 previous similar messages [ 7874.616338] Lustre: Skipped 7 previous similar messages [ 7877.500948] Lustre: server umount lustre-OST0000 complete [ 7887.784192] Lustre: server umount lustre-MDT0000 complete [ 7889.892323] LustreError: 264983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7889.913514] LustreError: 264983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7891.456461] LustreError: 264967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673459 with bad export cookie 8584790723625423288 [ 7891.466623] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7891.471176] LustreError: 264967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7891.999406] Lustre: server umount lustre-MDT0001 complete [ 7903.379912] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 7906.188898] Key type lgssc unregistered [ 7906.466603] LNet: 268707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7906.480042] LNetError: 268707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7906.500676] LNet: Removed LNI 192.168.206.133@tcp [ 7907.360153] Key type .llcrypt unregistered [ 7907.365471] Key type ._llcrypt unregistered [ 7936.340672] Key type ._llcrypt registered [ 7936.343282] Key type .llcrypt registered [ 7936.452589] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7937.820865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7937.850161] alg: No test for adler32 (adler32-zlib) [ 7938.969391] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7939.186275] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 7940.818148] Key type lgssc registered [ 7942.013764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7949.503683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7949.525489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7950.832314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.196339] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7961.059987] LustreError: 270166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7961.088318] LustreError: 270166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7964.027156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.690592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7969.667359] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7977.343219] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7983.096818] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7989.620431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7989.889674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.065081] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7996.026199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8001.683547] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8002.794618] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8006.631277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8006.638722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.649710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.763081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.769583] Lustre: Skipped 1 previous similar message [ 8010.879455] Lustre: server umount lustre-OST0000 complete [ 8014.817673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.830600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.838467] Lustre: Skipped 1 previous similar message [ 8014.854101] Lustre: Skipped 1 previous similar message [ 8019.944426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.817206] Lustre: server umount lustre-MDT0000 complete [ 8024.284247] LustreError: 270147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673592 with bad export cookie 4741103940800782481 [ 8024.289194] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.295540] LustreError: 270147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8024.739830] Lustre: server umount lustre-MDT0001 complete [ 8037.051821] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8040.316689] Key type lgssc unregistered [ 8040.620433] LNet: 273750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8040.629074] LNetError: 273750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8040.659369] LNet: Removed LNI 192.168.206.133@tcp [ 8041.705684] Key type .llcrypt unregistered [ 8041.708238] Key type ._llcrypt unregistered [ 8057.210983] Key type ._llcrypt registered [ 8057.213891] Key type .llcrypt registered [ 8057.353962] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8058.445521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8058.580363] alg: No test for adler32 (adler32-zlib) [ 8059.767960] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8060.045331] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8061.768328] Key type lgssc registered [ 8063.043638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8071.874339] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8071.905782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8087.520873] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8092.643325] LustreError: 274938:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.133@tcp: failed processing log, type 4: rc = -110 [ 8119.328677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.892870] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8131.184979] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8133.293098] Lustre: server umount lustre-OST0000 complete [ 8148.026721] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8150.463690] Key type lgssc unregistered [ 8150.756669] LNet: 276567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8150.765796] LNetError: 276567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8150.791670] LNet: Removed LNI 192.168.206.133@tcp [ 8151.574931] Key type .llcrypt unregistered [ 8151.578641] Key type ._llcrypt unregistered [ 8184.860261] Key type ._llcrypt registered [ 8184.864538] Key type .llcrypt registered [ 8184.942802] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8185.680553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8185.726108] alg: No test for adler32 (adler32-zlib) [ 8186.710221] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8186.951780] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8188.585167] Key type lgssc registered [ 8189.490688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8200.345527] loop0: detected capacity change from 0 to 204800000 [ 8200.564717] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8200.588598] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8200.939439] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8201.853239] loop0: detected capacity change from 0 to 204800000 [ 8202.061214] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8202.072684] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8202.378089] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8203.230565] loop0: detected capacity change from 0 to 204800000 [ 8203.362353] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8203.373905] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8203.447676] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8204.515137] loop0: detected capacity change from 0 to 204800000 [ 8204.679663] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8205.714509] loop0: detected capacity change from 0 to 204800000 [ 8205.740163] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8210.270280] loop0: detected capacity change from 0 to 204800000 [ 8210.304477] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8214.352952] loop0: detected capacity change from 0 to 204800000 [ 8214.381864] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8218.209967] loop0: detected capacity change from 0 to 204800000 [ 8218.250453] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8222.270054] loop0: detected capacity change from 0 to 204800000 [ 8222.330472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8222.359952] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8222.416830] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8222.426390] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8222.432783] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8222.439478] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8222.446409] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8222.453202] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8222.461159] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8222.467886] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8222.480341] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8222.498718] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8222.512640] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8222.530502] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8222.539387] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8222.548226] Lustre: 279978:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8228.443868] Lustre: server umount MGS complete [ 8229.340238] loop0: detected capacity change from 0 to 204800000 [ 8229.462076] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.102800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.147406] Lustre: lustre-MDT0000: Aborting client recovery [ 8231.149152] loop1: detected capacity change from 0 to 204800000 [ 8231.287719] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.359532] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8231.393490] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8231.430458] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8231.448880] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8231.469373] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8231.477845] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8231.485123] Lustre: 280408:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8231.519256] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8231.780255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8231.872457] Lustre: lustre-MDT0001: Aborting client recovery [ 8232.719265] loop2: detected capacity change from 0 to 204800000 [ 8232.839575] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8232.982936] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8233.014219] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8233.039549] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8233.051602] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8233.064943] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8233.081762] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8233.095698] Lustre: 280487:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8233.239283] Lustre: Found index 0 for lustre-OST0000, updating log [ 8233.298061] Lustre: 280505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8233.383906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.426402] Lustre: lustre-OST0000: Aborting recovery [ 8234.407767] loop3: detected capacity change from 0 to 204800000 [ 8234.623721] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8234.686533] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8234.705192] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8234.726247] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8234.738168] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8234.744986] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8234.752800] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8234.762230] Lustre: 280571:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8234.790907] Lustre: Found index 1 for lustre-OST0001, updating log [ 8234.819330] Lustre: 280576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8234.901676] Lustre: lustre-OST0001: Aborting recovery [ 8240.042354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8240.048927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8240.067312] LustreError: 280352:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8240.132243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8240.182354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8242.088451] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8242.088706] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8244.315539] LustreError: 280319:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 537us: rc = -2 [ 8244.827569] LustreError: 280317:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 398us: rc = -2 [ 8244.849452] LustreError: 280317:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 8257.505445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8257.506801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.521096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8257.523848] Lustre: Skipped 2 previous similar messages [ 8257.534862] Lustre: Skipped 3 previous similar messages [ 8262.625615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8262.631076] Lustre: Skipped 2 previous similar messages [ 8263.749604] Lustre: server umount lustre-MDT0000 complete [ 8264.586213] LustreError: 280304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673832 with bad export cookie 8028622525328351619 [ 8264.599312] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8264.607376] LustreError: 280304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8267.745914] LustreError: 280317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.751200] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.759604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8267.766556] LustreError: 280317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8267.790666] Lustre: Skipped 2 previous similar messages [ 8269.797183] LustreError: 280316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.800566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8269.826832] Lustre: Skipped 2 previous similar messages [ 8270.889985] Lustre: server umount lustre-MDT0001 complete [ 8277.803858] Lustre: server umount lustre-OST0000 complete [ 8284.825772] Lustre: server umount lustre-OST0001 complete [ 8293.262557] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 01:50:59 (1788673859) [ 8319.148885] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 8329.951149] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8365.826873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.106246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8366.157746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8366.284085] Lustre: lustre-MDT0000: new disk, initializing [ 8366.344720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.350571] Lustre: Skipped 1 previous similar message [ 8366.361803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8370.040976] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8380.074209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.191646] Lustre: 284642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8380.236155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8380.238884] Lustre: Skipped 1 previous similar message [ 8380.321376] Lustre: lustre-MDT0001: new disk, initializing [ 8380.398587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8380.446954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8380.462321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8384.552457] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8393.908604] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8399.448971] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8405.988053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8406.284893] Lustre: lustre-OST0000: new disk, initializing [ 8406.289673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8406.296640] Lustre: 286611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8406.387399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.345921] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8414.259652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8414.268072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8414.362215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8420.638342] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8425.496494] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8436.035280] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8442.017255] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8491.489358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8491.496719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.503319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8491.536963] Lustre: Skipped 1 previous similar message [ 8497.780905] Lustre: server umount lustre-OST0000 complete [ 8503.265643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8503.269145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.290982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8503.298019] Lustre: Skipped 3 previous similar messages [ 8508.250916] Lustre: server umount lustre-MDT0000 complete [ 8511.970456] LustreError: 284655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8511.994106] LustreError: 284655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8512.045954] LustreError: 288061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674079 with bad export cookie 8028622525328354328 [ 8512.050465] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8512.058090] LustreError: 288061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8512.495595] Lustre: server umount lustre-MDT0001 complete [ 8521.834770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.386367] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8531.254127] Lustre: server umount MGS complete [ 8542.406642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.920911] LustreError: 290480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.953597] LustreError: 290493:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8543.025766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8547.411678] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8548.321723] LustreError: 290481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.344324] LustreError: 290481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8556.365693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.248401] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8568.686457] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8573.958268] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8579.994724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.415393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.422401] Lustre: Skipped 1 previous similar message [ 8586.335835] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8593.776308] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8617.442944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8617.451447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.475097] Lustre: Skipped 1 previous similar message [ 8617.484743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8617.496739] Lustre: Skipped 2 previous similar messages [ 8623.514255] Lustre: server umount lustre-OST0000 complete [ 8628.193163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8628.197434] LustreError: Skipped 1 previous similar message [ 8628.200372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.216018] Lustre: Skipped 1 previous similar message [ 8633.570120] Lustre: server umount lustre-MDT0000 complete [ 8633.825319] LustreError: 290481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8633.844948] LustreError: 290481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8637.078688] LustreError: 290461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674205 with bad export cookie 8028622525328356155 [ 8637.084815] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8637.088348] LustreError: 290461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8637.474881] Lustre: server umount lustre-MDT0001 complete [ 8647.936754] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8650.913364] Key type lgssc unregistered [ 8651.314558] LNet: 294421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8651.318385] LNetError: 294421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8651.325826] LNet: Removed LNI 192.168.206.133@tcp [ 8652.216167] Key type .llcrypt unregistered [ 8652.217743] Key type ._llcrypt unregistered [ 8690.346242] Key type ._llcrypt registered [ 8690.348716] Key type .llcrypt registered [ 8690.467452] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 8704.661362] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8705.514562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8705.541542] alg: No test for adler32 (adler32-zlib) [ 8706.814163] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8707.076038] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8708.744233] Key type lgssc registered [ 8709.941933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8747.206410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8747.240100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.427891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8748.460294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8748.535764] Lustre: lustre-MDT0000: new disk, initializing [ 8748.593685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.611731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8752.806298] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8764.033567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.153382] Lustre: 298374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8764.202848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8764.207298] Lustre: Skipped 1 previous similar message [ 8764.267269] Lustre: lustre-MDT0001: new disk, initializing [ 8764.354901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8764.383277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8764.394540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8768.217206] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8777.425334] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8783.785945] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8790.783581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8791.059560] Lustre: lustre-OST0000: new disk, initializing [ 8791.064395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8791.068649] Lustre: 300342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8791.152528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.448618] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8800.811461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8800.839384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8800.974722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8807.246895] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8814.697141] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8827.126250] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8836.579690] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8843.118542] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8893.423207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8893.426136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.445504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8893.472557] Lustre: Skipped 1 previous similar message [ 8895.619413] Lustre: server umount lustre-OST0000 complete [ 8902.629648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8902.639845] LustreError: Skipped 1 previous similar message [ 8902.645992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.670715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.676390] Lustre: Skipped 1 previous similar message [ 8903.653774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.677585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8903.684827] Lustre: Skipped 1 previous similar message [ 8905.559367] Lustre: server umount lustre-MDT0000 complete [ 8908.808236] LustreError: 298367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674476 with bad export cookie 2844782201420665588 [ 8908.815303] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8908.818648] LustreError: 298367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8909.116182] Lustre: server umount lustre-MDT0001 complete [ 8917.199679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8921.841965] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8926.402276] Lustre: server umount MGS complete [ 8936.919480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.299801] LustreError: 304371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.314248] LustreError: 304371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8937.330136] LustreError: 304384:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8937.402287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.944933] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8942.568056] LustreError: 304372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.683689] LustreError: 304371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.264711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.620632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8953.287945] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8960.614486] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8965.553783] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8972.672742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8973.139213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8978.837325] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8985.448149] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9013.226601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.231294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.235661] Lustre: Skipped 1 previous similar message [ 9013.238476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9013.239295] Lustre: Skipped 1 previous similar message [ 9018.344400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9027.040163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9027.239570] Lustre: server umount lustre-OST0000 complete [ 9033.698051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9033.701197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9033.711158] Lustre: Skipped 1 previous similar message [ 9033.720140] Lustre: Skipped 4 previous similar messages [ 9037.845397] Lustre: server umount lustre-MDT0000 complete [ 9038.819659] LustreError: 304372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.839528] LustreError: 304372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9042.038232] LustreError: 304351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674609 with bad export cookie 2844782201420667534 [ 9042.051518] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9042.062792] LustreError: 304351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9042.504238] Lustre: server umount lustre-MDT0001 complete [ 9052.813972] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 9055.314424] Key type lgssc unregistered [ 9055.605129] LNet: 308317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9055.617205] LNetError: 308317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9055.633854] LNet: Removed LNI 192.168.206.133@tcp [ 9056.632613] Key type .llcrypt unregistered [ 9056.634653] Key type ._llcrypt unregistered [ 9076.967509] Key type ._llcrypt registered [ 9076.974738] Key type .llcrypt registered [ 9077.079320] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 9089.538436] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 9091.028462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9091.047684] alg: No test for adler32 (adler32-zlib) [ 9092.053312] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 9092.202932] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 9093.865062] Key type lgssc registered [ 9094.883458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9120.034486] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 9135.087813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9135.128070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.401962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9136.442571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9136.529936] Lustre: lustre-MDT0000: new disk, initializing [ 9136.632180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.655433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9140.261536] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9149.961764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9150.362162] Lustre: lustre-OST0000: new disk, initializing [ 9150.363298] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9150.365455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9150.376927] Lustre: Skipped 1 previous similar message [ 9150.383287] Lustre: 312346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9150.458250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9155.671707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9155.683443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9155.730644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9156.607929] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9990.114048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.130986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.064471] Lustre: server umount lustre-MDT0000 complete [ 9998.462237] LustreError: 311531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675566 with bad export cookie 6597000760122063017 [ 9998.472227] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9998.484717] LustreError: 311531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9998.683879] Lustre: server umount lustre-OST0000 complete [10025.368518] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [10037.430858] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10076.081414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.326514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10076.378162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10076.477192] Lustre: lustre-MDT0000: new disk, initializing [10076.568941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10076.589820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10080.889677] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10091.670915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.852675] Lustre: 316681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10091.944455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10091.951933] Lustre: Skipped 1 previous similar message [10092.090722] Lustre: lustre-MDT0001: new disk, initializing [10092.176928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10092.223722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10092.242652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10096.781729] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10107.438883] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10114.113431] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10122.790786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10123.138260] Lustre: lustre-OST0000: new disk, initializing [10123.141957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10123.149380] Lustre: 318648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10123.295885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10131.534993] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10132.598890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10132.610630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10132.811908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10142.848140] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10148.209191] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10148.494252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10154.363072] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10154.557982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10158.566877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10158.573248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.583509] LustreError: Skipped 1 previous similar message [10158.586364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.615782] Lustre: Skipped 1 previous similar message [10162.683092] Lustre: server umount lustre-OST0000 complete [10168.807225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.841258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.856351] Lustre: Skipped 2 previous similar messages [10172.067970] Lustre: server umount lustre-MDT0000 complete [10177.101972] LustreError: 316675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675745 with bad export cookie 6597000760125003836 [10177.104121] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10177.111912] LustreError: 316675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10177.623080] Lustre: server umount lustre-MDT0001 complete [10181.456396] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10183.609588] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 02:22:29 (1788675749) [10191.053758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.693151] LustreError: 320961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.715254] LustreError: 320961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10191.765595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10196.969882] LustreError: 320961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.312377] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10202.087881] LustreError: 320962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.206618] LustreError: 320961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.308721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10207.812178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10211.962824] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10219.087093] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10224.735972] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10232.268748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10232.632867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10240.792071] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10249.183397] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10256.896072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.034268] Lustre: lustre-OST0001: new disk, initializing [10257.036358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10257.041538] Lustre: 323798:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10257.118515] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10262.960815] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10263.121734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10263.137742] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10263.187527] LustreError: 324162:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10264.238461] LustreError: 324169:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10264.247045] LustreError: 324169:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10265.254053] LustreError: 324162:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10267.303495] LustreError: 324162:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10267.318907] LustreError: 324162:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10271.473388] LustreError: 324169:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10271.481763] LustreError: 324169:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10273.159995] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10273.401565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10279.307628] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d2342fbb800.ost_server_uuid 50 [10279.721995] LustreError: 324169:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10279.737053] LustreError: 324169:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10280.924730] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d2342fbb800.ost_server_uuid in FULL state after 0 sec [10287.172965] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d2342fbb800.ost_server_uuid 50 [10288.783373] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d2342fbb800.ost_server_uuid in FULL state after 0 sec [10291.128237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10303.881200] Lustre: lustre-OST0001: Not available for connect from 192.168.206.33@tcp (stopping) [10304.491726] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.506241] Lustre: Skipped 2 previous similar messages [10309.630259] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10309.647697] Lustre: Skipped 4 previous similar messages [10309.927654] Lustre: server umount lustre-OST0001 complete [10312.046308] LustreError: 322780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.821226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10318.831646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.872877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10319.844319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10322.822847] Lustre: server umount lustre-OST0000 complete [10330.084963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10330.098944] Lustre: Skipped 2 previous similar messages [10332.903691] Lustre: server umount lustre-MDT0000 complete [10335.203802] LustreError: 323257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.225209] LustreError: 323257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10336.348721] LustreError: 325332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675904 with bad export cookie 6597000760125004718 [10336.361534] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10336.368445] LustreError: 325332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10336.760588] Lustre: server umount lustre-MDT0001 complete [10347.652597] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [10350.544967] Key type lgssc unregistered [10350.951089] LNet: 326488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10350.980135] LNetError: 326488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10351.000685] LNet: Removed LNI 192.168.206.133@tcp [10351.949760] Key type .llcrypt unregistered [10351.952367] Key type ._llcrypt unregistered [10377.665986] Key type ._llcrypt registered [10377.676784] Key type .llcrypt registered [10377.818892] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10378.623606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10378.738168] alg: No test for adler32 (adler32-zlib) [10379.929858] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10380.163597] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [10381.880181] Key type lgssc registered [10382.999407] Lustre: Echo OBD driver; http://www.lustre.org/ [10390.515465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10390.534651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.948308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10395.894590] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10401.251553] LustreError: 327922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.265196] LustreError: 327922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10402.645413] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10406.372219] LustreError: 327923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.149841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.519929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10413.552214] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10422.136473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10422.360227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10429.412091] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10434.671515] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.718723] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.748327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10436.765549] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10438.135131] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10442.226774] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10447.076436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.307530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10452.499218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10455.609648] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10465.687649] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10473.265517] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10473.564698] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10580.463928] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10580.472702] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.482910] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10580.486467] Lustre: Skipped 1 previous similar message [10584.449430] Lustre: server umount lustre-OST0001 complete [10585.580079] LustreError: 329603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.618492] LustreError: 329603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10590.693472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10590.701564] LustreError: Skipped 1 previous similar message [10590.708932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10590.731975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10590.739744] Lustre: Skipped 2 previous similar messages [10595.811843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10596.548256] Lustre: server umount lustre-OST0000 complete [10600.930387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10600.932582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10600.945192] Lustre: Skipped 2 previous similar messages [10600.966696] Lustre: Skipped 2 previous similar messages [10606.050246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10606.064511] Lustre: Skipped 1 previous similar message [10606.753596] Lustre: server umount lustre-MDT0000 complete [10610.845040] LustreError: 331614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676178 with bad export cookie 10992933510117831315 [10610.851428] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10610.859037] LustreError: 331614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10611.283296] Lustre: server umount lustre-MDT0001 complete [10624.238664] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [10626.975587] Key type lgssc unregistered [10627.259881] LNet: 332867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10627.271177] LNetError: 332867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10627.288414] LNet: Removed LNI 192.168.206.133@tcp [10628.167688] Key type .llcrypt unregistered [10628.176151] Key type ._llcrypt unregistered [10670.268290] Key type ._llcrypt registered [10670.270428] Key type .llcrypt registered [10670.365469] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10671.294757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10671.565937] alg: No test for adler32 (adler32-zlib) [10672.553253] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10672.826042] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [10674.466154] Key type lgssc registered [10675.488342] Lustre: Echo OBD driver; http://www.lustre.org/ [10684.617517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10684.651173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.092790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10690.068355] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10695.651210] LustreError: 335155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.665455] LustreError: 335155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10697.981095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.252206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10702.464101] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10704.757217] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10711.203583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10711.427484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10717.678947] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10723.691508] LustreError: 336644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.919681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.103482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10730.230596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10730.246783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10730.930435] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10739.047835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10747.171830] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10759.651388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10759.665963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.675367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.192937] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.201057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.204826] Lustre: Skipped 2 previous similar messages [10761.219705] Lustre: Skipped 3 previous similar messages [10763.720199] Lustre: server umount lustre-MDT0000 complete [10766.310921] LustreError: 338021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.333572] LustreError: 338021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10766.724225] LustreError: 337466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676334 with bad export cookie 6165216976260528694 [10766.725932] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.733871] LustreError: 337466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10767.054950] Lustre: server umount lustre-MDT0001 complete [10780.770833] Lustre: server umount lustre-OST0000 complete [10793.388860] Lustre: server umount lustre-OST0001 complete [10807.744674] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10816.909913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.286558] LustreError: 340418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.382633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10821.311885] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10822.626167] LustreError: 340419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.275893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.647683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10832.642682] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10834.970668] Lustre: 341558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10840.875685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.232823] LustreError: 341911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.257915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10842.266730] LustreError: 341911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10847.089850] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10855.079989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10855.315643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10855.319903] Lustre: Skipped 1 previous similar message [10860.538166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10861.453706] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10868.413189] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10872.134475] Lustre: 343429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10879.970637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10879.988497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10880.006177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10882.600359] Lustre: server umount lustre-MDT0000 complete [10885.977769] LustreError: 340398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676453 with bad export cookie 6165216976260530395 [10885.978847] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10885.990665] LustreError: 340398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10886.114091] LustreError: 340415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.120722] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.125077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10886.125084] Lustre: Skipped 4 previous similar messages [10886.127012] LustreError: 340415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10886.162213] Lustre: Skipped 4 previous similar messages [10886.365727] Lustre: server umount lustre-MDT0001 complete [10892.171386] Lustre: server umount lustre-OST0000 complete [10895.711353] Lustre: server umount lustre-OST0001 complete [10899.128695] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10900.563729] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10902.156417] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 02:34:28 (1788676468) [10922.724391] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10931.482390] loop0: detected capacity change from 0 to 1099511627776 [10932.074812] SGI XFS with ACLs, security attributes, quota, no debug enabled [10932.087271] *** VALIDATE xfs *** [10932.101367] XFS (loop0): Mounting V5 Filesystem [10932.127655] XFS (loop0): Ending clean mount [10933.384891] loop1: detected capacity change from 0 to 18691697672192 [11026.489638] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [11035.639598] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [11074.137157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11074.599308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11074.640769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11074.771667] Lustre: lustre-MDT0000: new disk, initializing [11074.970125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11075.027754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11081.627483] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11093.236786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11093.360782] Lustre: 348559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11093.411760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11093.416273] Lustre: Skipped 1 previous similar message [11093.502983] Lustre: lustre-MDT0001: new disk, initializing [11093.657118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11093.674490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11097.677816] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11106.686486] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11111.979174] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11117.989437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11118.289463] Lustre: lustre-OST0000: new disk, initializing [11118.297784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11118.315975] Lustre: 350528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11119.948858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11119.959075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11120.054968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11124.292803] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11132.934781] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11137.007042] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11137.214494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11142.076751] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11142.375432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11145.701898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11145.705430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.708719] LustreError: Skipped 1 previous similar message [11145.709850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11145.720232] Lustre: Skipped 1 previous similar message [11145.737053] Lustre: Skipped 2 previous similar messages [11150.535538] Lustre: server umount lustre-OST0000 complete [11154.914794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11154.929704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.948155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11160.785424] Lustre: server umount lustre-MDT0000 complete [11161.059829] LustreError: 348571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.085030] LustreError: 348571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11164.708986] LustreError: 348552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676732 with bad export cookie 6165216976260531949 [11164.719701] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.723765] LustreError: 348552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11165.195322] Lustre: server umount lustre-MDT0001 complete [11174.207374] XFS (loop0): Unmounting Filesystem [11178.517177] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 02:39:04 (1788676744) [11186.122336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.485467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.493336] Lustre: Skipped 2 previous similar messages [11191.648429] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11202.587598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11209.016613] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11216.564793] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.458481] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11230.175502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11236.583239] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11244.052720] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11258.339290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11258.346672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11258.362351] Lustre: Skipped 1 previous similar message [11258.370050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11258.372370] Lustre: Skipped 2 previous similar messages [11261.683589] Lustre: server umount lustre-OST0000 complete [11271.739487] Lustre: server umount lustre-MDT0000 complete [11275.355949] LustreError: 353065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676843 with bad export cookie 6165216976260532824 [11275.362044] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.369755] LustreError: 353065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11275.819951] Lustre: server umount lustre-MDT0001 complete [11287.237799] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [11289.992986] Key type lgssc unregistered [11290.362791] LNet: 356715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11290.382732] LNetError: 356715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11290.403792] LNet: Removed LNI 192.168.206.133@tcp [11291.213501] Key type .llcrypt unregistered [11291.215960] Key type ._llcrypt unregistered [11318.591381] Key type ._llcrypt registered [11318.596492] Key type .llcrypt registered [11318.704252] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [11319.866439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11319.935444] alg: No test for adler32 (adler32-zlib) [11321.034807] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [11321.291221] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [11322.976345] Key type lgssc registered [11323.907928] Lustre: Echo OBD driver; http://www.lustre.org/ [11332.199715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11332.221697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11333.748375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11337.867387] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11343.333765] LustreError: 358282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.358902] LustreError: 358282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11346.345967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.764386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11351.425745] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11359.331805] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11365.108838] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11371.720396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.963293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11378.692164] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11385.897518] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11396.584014] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.595442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11396.600226] Lustre: Skipped 1 previous similar message [11398.000935] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.33@tcp (stopping) [11401.704236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11401.715387] Lustre: Skipped 1 previous similar message [11401.879366] Lustre: server umount lustre-MDT0001 complete [11403.100291] LustreError: 358279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.819183] LustreError: 358277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.850891] LustreError: 358277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11411.937645] LustreError: 358282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11411.978703] LustreError: 358282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11412.475404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11412.575268] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11412.578280] Lustre: Found index 1 for lustre-MDT0001, updating log [11412.617493] Lustre: 358270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11413.042575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11418.099320] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11418.590450] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11418.658655] LustreError: 361732:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11418.664520] LustreError: 361732:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11418.690937] LustreError: 358274:0:(mgc_request.c:612:do_requeue()) failed processing log: -17 [11418.800567] LustreError: 361739:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.133@tcp: cfg command failed: rc = -17 [11418.840092] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11418.840092] [11433.446107] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11433.446918] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.466056] Lustre: Skipped 1 previous similar message [11433.478252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11433.488078] Lustre: Skipped 1 previous similar message [11436.322743] Lustre: server umount lustre-MDT0001 complete [11438.587765] LustreError: 360774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.623785] LustreError: 360774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11446.242255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.387962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11446.399700] Lustre: Found index 1 for lustre-MDT0001, updating log [11446.461983] Lustre: 358270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11447.054300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11450.891297] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11452.393476] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11454.450465] LustreError: 362958:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11454.459092] LustreError: 362958:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11454.473339] LustreError: 358274:0:(mgc_request.c:612:do_requeue()) failed processing log: -17 [11454.479368] LustreError: 358274:0:(mgc_request.c:612:do_requeue()) Skipped 1 previous similar message [11454.526619] LustreError: 362961:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11454.531326] LustreError: 362961:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11454.536731] LustreError: 362961:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.133@tcp: cfg command failed: rc = -17 [11454.546212] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.133@tcp [11454.546212] [11756.400504] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.33@tcp (stopping) [11756.406747] Lustre: Skipped 1 previous similar message [11759.598163] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11759.602133] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11759.615467] Lustre: Skipped 1 previous similar message [11764.706056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11764.712669] Lustre: Skipped 4 previous similar messages [11770.338516] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11770.660342] Lustre: server umount lustre-MDT0001 complete [11771.741708] LustreError: 358278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11771.759747] LustreError: 358278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11780.141061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11780.289828] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11780.293244] Lustre: Found index 1 for lustre-MDT0001, updating log [11780.374180] Lustre: 358272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11780.845396] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11784.944459] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11786.238864] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11789.820477] LustreError: 371982:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11789.847147] LustreError: 371982:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11789.860189] LustreError: 358274:0:(mgc_request.c:612:do_requeue()) failed processing log: -17 [11789.878594] LustreError: 358274:0:(mgc_request.c:612:do_requeue()) Skipped 1 previous similar message [11790.016379] LustreError: 371985:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11790.040210] LustreError: 371985:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.133@tcp: cfg command failed: rc = -114 [11790.050319] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11790.050319] [12108.769786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.785260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12108.786996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12108.787032] Lustre: Skipped 3 previous similar messages [12108.793865] Lustre: Skipped 3 previous similar messages [12110.097600] Lustre: server umount lustre-MDT0000 complete [12113.892666] LustreError: 365487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12113.909869] LustreError: 365487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12113.912641] LustreError: 358262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677681 with bad export cookie 11903971537966119104 [12113.915115] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12113.953679] LustreError: 358262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12119.014754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.225186] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12128.438569] Lustre: server umount lustre-MDT0001 complete [12132.042909] Lustre: server umount lustre-OST0000 complete [12140.832460] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [12149.869333] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [12186.631969] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 02:55:52 (1788677752) [12192.939941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.129391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12193.148751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12193.222187] Lustre: lustre-MDT0000: new disk, initializing [12193.300645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12193.325214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12197.245944] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12206.595174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.714989] Lustre: 383161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12206.817853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12206.821902] Lustre: Skipped 1 previous similar message [12206.919968] Lustre: lustre-MDT0001: new disk, initializing [12207.046265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12207.094826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12207.100500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12211.171301] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12220.737971] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12226.212297] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12232.273766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12232.517872] Lustre: lustre-OST0000: new disk, initializing [12232.524726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12232.539143] Lustre: 385130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12232.621416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12238.366593] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12238.915611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12238.925962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12239.062059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12248.605137] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12256.756526] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12256.765746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12256.777027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12256.784805] Lustre: Skipped 5 previous similar messages [12258.540394] Lustre: server umount lustre-MDT0000 complete [12259.823301] LustreError: 384005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12259.840861] LustreError: 384005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12262.245632] LustreError: 383956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677830 with bad export cookie 11903971537966121057 [12262.248655] LustreError: 383968:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12262.255131] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12262.261700] LustreError: 383956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12262.546243] Lustre: server umount lustre-MDT0001 complete [12276.558712] Lustre: server umount lustre-OST0000 complete [12306.793733] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [12316.046408] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [12342.578171] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12071 sec ========= 02:58:28 (1788677908) [12344.544889] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:58:30 (1788677910) === [12347.618328] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:58:33 (1788677913) === [12381.440766] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [12383.924524] Key type lgssc unregistered [12384.237758] LNet: 391493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12384.244675] LNetError: 391493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12384.261777] LNet: Removed LNI 192.168.206.133@tcp [12384.964266] Key type .llcrypt unregistered [12384.966866] Key type ._llcrypt unregistered